-- Logs begin at Sat 2022-08-27 01:35:59 UTC, end at Sat 2022-08-27 02:51:47 UTC. -- Aug 27 02:06:28.049059 dv-cinder-846303-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 27 02:06:28.058137 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sat Aug 27 02:06:28 2022] *** Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: nodename: dv-cinder-846303-3-iscsi Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: machine: x86_64 Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: clock source: unix Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: pcre jit disabled Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: detected number of CPU cores: 4 Aug 27 02:06:28.060496 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: current working directory: / Aug 27 02:06:28.061432 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: detected binary path: /usr/bin/uwsgi-core Aug 27 02:06:28.061432 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: your processes number limit is 47680 Aug 27 02:06:28.061432 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: your memory page size is 4096 bytes Aug 27 02:06:28.061432 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: detected max file descriptor number: 2048 Aug 27 02:06:28.061432 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: lock engine: pthread robust mutexes Aug 27 02:06:28.061432 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: thunder lock: enabled Aug 27 02:06:28.061432 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 27 02:06:28.065010 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 27 02:06:28.110689 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: Python main interpreter initialized at 0x55f20c3269c0 Aug 27 02:06:28.110689 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: python threads support enabled Aug 27 02:06:28.110689 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: your server socket listen backlog is limited to 100 connections Aug 27 02:06:28.110689 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: your mercy for graceful operations on workers is 90 seconds Aug 27 02:06:28.111066 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: mapped 403077 bytes (393 KB) for 2 cores Aug 27 02:06:28.111177 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: *** Operational MODE: preforking *** Aug 27 02:06:28.111484 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: *** uWSGI is running in multiple interpreter mode *** Aug 27 02:06:28.111484 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: spawned uWSGI master process (pid: 81731) Aug 27 02:06:28.111727 dv-cinder-846303-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 27 02:06:28.113070 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: spawned uWSGI worker 1 (pid: 81745, cores: 1) Aug 27 02:06:28.113070 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: spawned uWSGI worker 2 (pid: 81746, cores: 1) Aug 27 02:06:28.113070 dv-cinder-846303-3-iscsi devstack@keystone.service[81731]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 27 02:06:33.094927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 27 02:06:33.094927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: from cryptography.utils import int_from_bytes Aug 27 02:06:33.095778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 27 02:06:33.095778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: from cryptography.utils import int_from_bytes Aug 27 02:06:33.171773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 27 02:06:33.172068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 27 02:06:33.172382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] command line args: [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 27 02:06:33.172776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 27 02:06:33.173081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ================================================================================ {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 27 02:06:33.173508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] admin_token = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.173772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] config_dir = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.174082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.174396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] config_source = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.174726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] debug = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.175088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.176884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.176884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.176884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.176884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.177379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] list_limit = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.177600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] log_config_append = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.177916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.178187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] log_dir = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.178527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] log_file = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.178954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.179256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.179654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.179993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.180235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.180554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.181047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.181494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.181943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.182369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.182632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.182955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.183399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.183679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.183976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.184498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.52/identity {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.184686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] publish_errors = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.184932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.185203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.185394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.185651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.185904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.186139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] standard_threads = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.186385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.186636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.186814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.187031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] use_journal = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.187243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] use_json = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.187428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] use_stderr = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.187607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] use_syslog = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.187849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.188118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.188506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.188965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.189534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.189862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.190300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.190758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.191199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.191571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.191868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.192203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.192454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.192713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.192960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.193434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.193698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.193941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.194206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.194469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.194715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.194958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.195219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.195492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.195747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.196000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.196237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.196507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.196686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.196932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.197208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.197471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.197705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.197940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.198188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.198559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.198762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.199012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.199249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.199489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.199725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.199959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.200190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.200521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.200936 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.201386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.201836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.202213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.202668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.203099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.203451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.203861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.204195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.204512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.204810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.205183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.205570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.206017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.206344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.206688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.206955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.207202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.207633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.207875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.208257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.208506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.208800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.209150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.209441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.209868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.210302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.210733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.211187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.211622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.212031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.212469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.212890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.213361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.213695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.214062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.214449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.214911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.215247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.215598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.216051 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.216496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.217009 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.217483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.217894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.218356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.218779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.219220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.219682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.220142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.220591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.221050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.221530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.221977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.222671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.223047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.223478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.223929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.224241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.224679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.225110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.225618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.226083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.226505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.226935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] auth.external = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.227259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.227802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.228288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.228734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] auth.password = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.229046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] auth.token = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.229428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.229794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.230191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.230617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.231066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.231460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.231898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.232351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] credential.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.232808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.233335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.233729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.234142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.234584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.234975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.235440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.235922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.236489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.236839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.237417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.237733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.238208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.238592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.239181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.239395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.239630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.240070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.240532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.240931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.241416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.241858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.242330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.242774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.243249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.243700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.244161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.244621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.245099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.245603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.246058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.246502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.246892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.247248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.247680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.248089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.248484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.248862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.249215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.249636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.250319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.250463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.250812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.251315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.251792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.252306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.252762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.253413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.253693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.254243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.254696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.255096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.255551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.255962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.256437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.256880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.257383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.257853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.258169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.258593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.258990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.259418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.259825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.260188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.260629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.261074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.261730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.262054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.262442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.262769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.263039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.263470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.263878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.264333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.264800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.265267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.265705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.265976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.266229 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.266484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.266758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.267006 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.267265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.267523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.267778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.268018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.268365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.268798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.269243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.269693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.270709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.270709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.271089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.271485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.271726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.272019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.272360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.272630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.272973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.273250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.273562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.274044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.274317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.274714 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.275153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.275557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.275879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.276243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.276675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.276951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.277330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.277781 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.278085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.278427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.278689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.278939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.279270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.279543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.279772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.280063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.280337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.280578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.280833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.281076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.281373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.281799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] role.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.282240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] role.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.282492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.282888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.283191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.283495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.283878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.284252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.284676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.284971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.285267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.285576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.286147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.286433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.286946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.287315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.287865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.288113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.288378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.288669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.288930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.289379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.289807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.290233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.290498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.290823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.291244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.291539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.291902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.292211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.292596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.293041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.293498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.297305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.298809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.298809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.298809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.298809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.298809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.298809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.299653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.299653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.299943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.300274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.300633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.301100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.connection = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.301642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.301917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.302173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.302456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.302875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.303303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.303724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.304684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.304684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.305422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.305422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.306532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.306532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.306532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.309062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.309062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.309062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.309062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.309062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81746) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 27 02:06:33.356165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 27 02:06:33.356165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: from cryptography.utils import int_from_bytes Aug 27 02:06:33.356997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 27 02:06:33.356997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: from cryptography.utils import int_from_bytes Aug 27 02:06:33.357668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.358056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.358449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.373943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.413091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 27 02:06:33.413564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 27 02:06:33.413826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] command line args: [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 27 02:06:33.414139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 27 02:06:33.414438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ================================================================================ {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 27 02:06:33.416197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] admin_token = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.416197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] config_dir = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.416197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.416197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] config_source = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.416197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] debug = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.417222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.417222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.417222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.417826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.417826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.418143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.418317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] list_limit = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.418482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.418697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] log_config_append = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.418866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.419029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] log_dir = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.419351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] log_file = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.419816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.420262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.420687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.421108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.421488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.421825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.422253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.422746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.423210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.423694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.424167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.424736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.425187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.425647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.426172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.427048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.52/identity {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.427391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] publish_errors = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.427829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.428261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.428595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.429063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.429511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.429767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] standard_threads = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.430012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.430270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.430453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.430636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] use_journal = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.430852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] use_json = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.431103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] use_stderr = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.431407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] use_syslog = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.431828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 27 02:06:33.432303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.432634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.432955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.433452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.433926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.434384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.434848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.435295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.435714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.436124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.436559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.436972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.437423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.437872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.438370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.438792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.439133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.439572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.439996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.440484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.440884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.441383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.442053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.442710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.443206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.443775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.444198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.444523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.444909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.445335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.445669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.445986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.446444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.446741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.447002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.447598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.448078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.448558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.449026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.452779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.454039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.454039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.454039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.454399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.454740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.455000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.455254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.455542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.455798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.456355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.456607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.456867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.457150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.457414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.457655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.457928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.458169 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.458405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.458647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.458896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.459179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.459420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.459673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.459926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.460159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.460401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.460638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.460886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.461186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.461394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.461634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.461876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.462128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.462363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.462604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.462840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.463077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.463311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.463576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.463815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.464063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.464309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.464550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.464793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.465030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.465296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.465533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.465763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.465995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.466224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.466468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.466885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.467051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.467313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.467547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.467785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.468044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.468306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.468594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.468883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] auth.external = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.469200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.469519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.469848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.470204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] auth.password = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.470526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] auth.token = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.470825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.471152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.471499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.471779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.472468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.472873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.473349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.473718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] credential.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.474062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.474673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.475096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.475569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.475856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.476099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.476355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.476611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.476855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.477163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.477583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.478018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.478480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.478890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.479378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.479797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.480221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.480648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.481080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.481532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.481972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.482403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.482840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.483261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.483698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.484125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.484566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.484996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.485461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.485911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.486329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.486757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.487243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.487700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.488158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.489430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.490005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.490311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.490728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.491149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.491599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.492042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.492459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.492912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.493387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.493596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.493876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.494378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.494802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.495227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.495662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.496092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.496573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.497016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.497490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.497985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.498411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.498895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.499365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.499816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.500234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.500599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.500962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.501427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.501752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.502088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.502362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.502644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.502940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.503257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.503580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.503935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.504234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.504597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.504944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.505411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.505814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.506184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.506606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.507014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.507304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.507592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.507865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.508148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.508434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.508771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.509047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.509339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.509601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.509865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.510138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.510402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.510688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.510961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.511243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.511524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.511951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.512341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.512746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.513261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.513626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.513970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.514335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.514736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.515130 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.515513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.515893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.516221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.516612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.517026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.517434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.517787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.518204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.518645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.519032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.519450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.520324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.520792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.521262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.521654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.521946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.522240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.522486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.522745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.523022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.523273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.523556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] role.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.523809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] role.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.524048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.524302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.524559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.524807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.525089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.525369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.525620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.525868 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.526118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.526356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.526605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.526946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.527332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.527698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.528279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.528683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.529095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.529559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.530000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.530423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.530843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.531308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.531984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.532196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.532614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.533025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.533385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.533733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.534059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.534421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.534778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.535114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.535447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.535731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.535995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.536394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.536805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.537071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.537446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.537784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.538230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.538679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.539005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.539331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.539617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.539931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.540345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.540841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.541194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.541498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.541973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.542320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.connection = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.542675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.543006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.543331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.543639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.543948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.544267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.544578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.544825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.545247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.545695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.546159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.546640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.547114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.547563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.548042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.548478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.548933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.549415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 27 02:06:33.549733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81745) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 27 02:06:33.558832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.559165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.581757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.582021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.582217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.597689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.623435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.623753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.672493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.724192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.724566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.752956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.753361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.753572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.753774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.754001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.754391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.780875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.781181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.781457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.781716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.781977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.782124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.782260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.782388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.782528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.782653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.782860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.783141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.783269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.796424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.796666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.796876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.797182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.797414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.797635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.797849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.798066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.812534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.814942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.815180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.815412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.815688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.816041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.816420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.816862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81746) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.840527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.840923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 27 02:06:33.841065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.841304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 27 02:06:33.841784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.842200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.842561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.842836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.855962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.856269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.856590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.857067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.867984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.871375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.871673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.871965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.872256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.872549 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.872814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.873088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.873403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.873743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.873959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.874235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.897848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.899396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.899682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.905592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.906116 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.906594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.914421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.914744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.918932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.919237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.919533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.919825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.920108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.922396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.922701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.922997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.931849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.932185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.932509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.932833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.933171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.933486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.933794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81745) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 27 02:06:33.939373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.939667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.943741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.946960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.947275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.948363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.948670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 27 02:06:33.949274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.949622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 27 02:06:33.950126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.950590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.951054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.951521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.951763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.954964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.958298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.958611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.958901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.959192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.959474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.959890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.964884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.965386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.965816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.966263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.974836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.976521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.977017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.977510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.977953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.980018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.980532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.981003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.981524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.982029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.982492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.982947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.983425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.983914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.984362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.984835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.985905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.987631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.989346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.989813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.994042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:33.994530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.995017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.995523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:33.996014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.011425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.011739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.012032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.012325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.012632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.012920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.013244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.022130 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.024882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.025405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.029242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.032829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.033750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.034126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.034555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.035846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.037407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.038891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.039186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.039474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.045383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.045697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.046734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.049025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.050113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.050586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.051057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.051411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.051748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.052082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.052424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.052816 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.052979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.053109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.053423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.053791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.062653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.062974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.063271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.074698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81746) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.075182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.075613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.076041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.076447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.076795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.077103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.077244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.077423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.077622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.077738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.078001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.078439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.078900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.078900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.079370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81746) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.089078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.089565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.096274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.101384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.106200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.106608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.106994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.107374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.107757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.108177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.118014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55f20c3269c0 pid: 81746 (default app) Aug 27 02:06:34.121836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba2d6e6-ac0d-45c4-8460-90a734a5381e None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:34.122128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba2d6e6-ac0d-45c4-8460-90a734a5381e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:34.122397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba2d6e6-ac0d-45c4-8460-90a734a5381e None None] PATH_INFO: `/v3` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:34.126748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1/1] 10.222.0.52 () {50 vars in 791 bytes} [Sat Aug 27 02:06:34 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:34.132351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.135773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.135773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.135773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.142768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.146105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.147760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.148049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.150734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.151670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.151670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.151670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.151969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.165084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.165601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.166039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.166490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.166965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.167419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.167887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.193579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.197719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.202284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.204964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.207753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.208544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.208979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.218792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.220359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.223857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.224160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.224489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.224817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.233348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.233664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.233947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.242103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81745) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 27 02:06:34.242414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.242723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.243005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.243289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.243661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.243952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.244226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.244506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.244778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.245063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.245358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.245645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.245908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81745) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 27 02:06:34.293190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55f20c3269c0 pid: 81745 (default app) Aug 27 02:06:41.446634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d27e206-41f0-4552-9184-cccf9528baf6 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:41.447291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d27e206-41f0-4552-9184-cccf9528baf6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:41.447291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d27e206-41f0-4552-9184-cccf9528baf6 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:41.448613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 2/2] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:41.458332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:41.459138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:41.459138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:41.536938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: INFO dbcounter [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] Registered counter for database keystone Aug 27 02:06:41.559284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG oslo_db.sqlalchemy.engines [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81745) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 27 02:06:41.577091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writer thread running {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 27 02:06:41.799499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG passlib.handlers.bcrypt [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81745) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 27 02:06:41.799978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG passlib.handlers.bcrypt [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81745) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 27 02:06:41.858028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:41.859752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:42.037503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1/3] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:41 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 583 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Aug 27 02:06:42.631257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d27e206-41f0-4552-9184-cccf9528baf6 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:42.724096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: INFO dbcounter [None req-3d27e206-41f0-4552-9184-cccf9528baf6 None None] Registered counter for database keystone Aug 27 02:06:42.748412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG oslo_db.sqlalchemy.engines [None req-3d27e206-41f0-4552-9184-cccf9528baf6 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81746) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 27 02:06:42.756783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writer thread running {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 27 02:06:42.887957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d27e206-41f0-4552-9184-cccf9528baf6 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:42.893191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87297165-a027-4265-875e-ff98881049ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:42.893830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87297165-a027-4265-875e-ff98881049ad admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:42.894138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87297165-a027-4265-875e-ff98881049ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:42.894364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87297165-a027-4265-875e-ff98881049ad admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:42.904135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87297165-a027-4265-875e-ff98881049ad admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:42.956713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING keystone.common.rbac_enforcer.enforcer [None req-87297165-a027-4265-875e-ff98881049ad admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 27 02:06:42.957210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87297165-a027-4265-875e-ff98881049ad admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:42.968751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 3/4] 10.222.0.52 () {58 vars in 1324 bytes} [Sat Aug 27 02:06:42 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:06:42.975065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:42.991434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edc22ab-f80b-4c4d-840f-47192e18bbf4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:42.996135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966261af-7ef7-48dd-a941-ed26a56aa6bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:42.997461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966261af-7ef7-48dd-a941-ed26a56aa6bf admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:42.997745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966261af-7ef7-48dd-a941-ed26a56aa6bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:42.998018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966261af-7ef7-48dd-a941-ed26a56aa6bf admin admin] PATH_INFO: `/v3/projects/f1df9f889b3642a7ac056884bc2c51a4` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:43.000004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966261af-7ef7-48dd-a941-ed26a56aa6bf admin admin] RBAC: Authorizing `identity:get_project(project_id=f1df9f889b3642a7ac056884bc2c51a4)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:43.053359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING keystone.common.rbac_enforcer.enforcer [None req-966261af-7ef7-48dd-a941-ed26a56aa6bf admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 27 02:06:43.053796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966261af-7ef7-48dd-a941-ed26a56aa6bf admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:43.059021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 2/5] 10.222.0.52 () {56 vars in 1207 bytes} [Sat Aug 27 02:06:42 2022] GET /identity/v3/projects/f1df9f889b3642a7ac056884bc2c51a4 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:45.084711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242227b1-988c-4acd-af05-94f352bd4581 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:45.086117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242227b1-988c-4acd-af05-94f352bd4581 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:45.086117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242227b1-988c-4acd-af05-94f352bd4581 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:45.090025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 4/6] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:45.096401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4111215-0b17-4507-ad0d-d27f281e13cd None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:45.096917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4111215-0b17-4507-ad0d-d27f281e13cd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:45.096917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4111215-0b17-4507-ad0d-d27f281e13cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:45.127977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a4111215-0b17-4507-ad0d-d27f281e13cd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:45.129564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a4111215-0b17-4507-ad0d-d27f281e13cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:45.139815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 3/7] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:45 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 27 02:06:45.430181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242227b1-988c-4acd-af05-94f352bd4581 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:45.443975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242227b1-988c-4acd-af05-94f352bd4581 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:45.448482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05a6509-e8d4-4c5d-9d5e-341ea9ff8c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:45.449164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05a6509-e8d4-4c5d-9d5e-341ea9ff8c80 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:45.449441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05a6509-e8d4-4c5d-9d5e-341ea9ff8c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:45.449716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05a6509-e8d4-4c5d-9d5e-341ea9ff8c80 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:45.458376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05a6509-e8d4-4c5d-9d5e-341ea9ff8c80 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:45.460369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05a6509-e8d4-4c5d-9d5e-341ea9ff8c80 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:45.472264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 5/8] 10.222.0.52 () {58 vars in 1324 bytes} [Sat Aug 27 02:06:45 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:06:45.478257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4111215-0b17-4507-ad0d-d27f281e13cd None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:45.489195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4111215-0b17-4507-ad0d-d27f281e13cd None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:45.494057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed86e0ea-deaf-473d-b285-5feb57847492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:45.494720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86e0ea-deaf-473d-b285-5feb57847492 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:45.494997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86e0ea-deaf-473d-b285-5feb57847492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:45.495266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed86e0ea-deaf-473d-b285-5feb57847492 admin admin] PATH_INFO: `/v3/users/6648709a4b6d470c80e9b42cb0d345ad` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:45.496880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed86e0ea-deaf-473d-b285-5feb57847492 admin admin] RBAC: Authorizing `identity:get_user(user_id=6648709a4b6d470c80e9b42cb0d345ad)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:45.498784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed86e0ea-deaf-473d-b285-5feb57847492 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:45.500407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 4/9] 10.222.0.52 () {56 vars in 1198 bytes} [Sat Aug 27 02:06:45 2022] GET /identity/v3/users/6648709a4b6d470c80e9b42cb0d345ad => generated 247 bytes in 23 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:47.877068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aedbbb0-9faf-431a-9c6b-ba826f452e20 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:47.877735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aedbbb0-9faf-431a-9c6b-ba826f452e20 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:47.877735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aedbbb0-9faf-431a-9c6b-ba826f452e20 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:47.878974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 6/10] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:47.886717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161ee44-9625-48eb-bb36-f8019fee5a50 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:47.887248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161ee44-9625-48eb-bb36-f8019fee5a50 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:47.887360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161ee44-9625-48eb-bb36-f8019fee5a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:47.908280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b161ee44-9625-48eb-bb36-f8019fee5a50 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:47.909540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b161ee44-9625-48eb-bb36-f8019fee5a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:47.983830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb585e79-2834-407d-8eb3-42e285c6fd5d None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:47.984173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb585e79-2834-407d-8eb3-42e285c6fd5d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:47.984670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb585e79-2834-407d-8eb3-42e285c6fd5d None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:47.986915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 7/11] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:06:47.994972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:47.995552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:47.996120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:48.019500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 5/12] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:06:48.061918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG passlib.handlers.bcrypt [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81746) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 27 02:06:48.062415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG passlib.handlers.bcrypt [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81746) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 27 02:06:48.110388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:48.112767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:48.253155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 8/13] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:06:48.297692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161ee44-9625-48eb-bb36-f8019fee5a50 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:48.329069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:48.433355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161ee44-9625-48eb-bb36-f8019fee5a50 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:48.490455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:48.491359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:48.491721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:48.492005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:48.547438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:48.588371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:48.589916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:48.621243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:48.698822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951ebce8-6b43-465a-9e4c-0cdcf0b454ff None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:48.702501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:48.745623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 6/14] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:06:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 27 02:06:48.751637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:48.760618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:48.762665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:48.763140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:48.763567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:48.779510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:06:48.788778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:06:48.790171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:06:48.791328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 9/15] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:06:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:06:48.799367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:48.892947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfd32e7-44ec-472f-97a6-faa3b51e24b0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:48.926089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b236674-b8d0-4aed-ad2b-799b2be69a45 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:48.926948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:48.941780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:48.942520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:48.942856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:48.943182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:48.955679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:48.957489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:48.967931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:48.968377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:48.968676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:48.969036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:48.969948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:06:48.969948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:48.971419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:06:48.978343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:06:48.979734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:06:48.980769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:06:48.980769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:06:48.980769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:06:48.980769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 7/16] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:06:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:06:48.987120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 10/17] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:06:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:06:48.988083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:48.993269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:49.029990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:49.111702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182fd00c-09e8-4a5a-b323-9ae18d5a76cb None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:49.118410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb31c6b2-e06d-4571-88e0-e8b200b2dbe2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:49.159248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:49.159853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:49.160030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:49.160186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:49.160390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:49.160809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:49.161043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:06:49.161280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:49.161415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:49.161664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] PATH_INFO: `/v3/users/6648709a4b6d470c80e9b42cb0d345ad` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:49.162767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:49.163846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] RBAC: Authorizing `identity:get_user(user_id=6648709a4b6d470c80e9b42cb0d345ad)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:49.166034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:49.168288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 11/18] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:06:48 2022] GET /identity/v3/users/6648709a4b6d470c80e9b42cb0d345ad => generated 247 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:06:49.174154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:49.197149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 8/19] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:06:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:49.304708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:49.312977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e201744a-9477-4ece-81f3-6bb59aafb610 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:49.352176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964acd10-7f46-4906-a27d-cf0e5d2acc5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:49.353206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964acd10-7f46-4906-a27d-cf0e5d2acc5c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:49.353694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964acd10-7f46-4906-a27d-cf0e5d2acc5c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:49.354101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964acd10-7f46-4906-a27d-cf0e5d2acc5c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:49.355332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964acd10-7f46-4906-a27d-cf0e5d2acc5c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:49.356794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964acd10-7f46-4906-a27d-cf0e5d2acc5c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:49.358815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 12/20] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:06:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:06:49.365095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:49.414323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:49.516007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35c245c-2dc7-4b53-a73c-5abe8a80de90 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:49.568266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183ba772-f4c1-4833-ae51-bbdce4e990cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:49.569095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183ba772-f4c1-4833-ae51-bbdce4e990cd None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:49.569506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183ba772-f4c1-4833-ae51-bbdce4e990cd None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:49.569923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183ba772-f4c1-4833-ae51-bbdce4e990cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:49.570949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183ba772-f4c1-4833-ae51-bbdce4e990cd None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.domain.id=default, user.id=6648709a4b6d470c80e9b42cb0d345ad) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:06:49.571355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183ba772-f4c1-4833-ae51-bbdce4e990cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:49.573710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183ba772-f4c1-4833-ae51-bbdce4e990cd None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:49.586320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 9/21] 10.222.0.52 () {56 vars in 1322 bytes} [Sat Aug 27 02:06:49 2022] GET /identity/v3/role_assignments?user.id=6648709a4b6d470c80e9b42cb0d345ad&scope.domain.id=default&role.id=b83e461b8f6643fba45f427d28498c41 => generated 233 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:52.697882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f881a90-bf74-40b9-b82c-6c4c3c058bc2 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:52.698676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f881a90-bf74-40b9-b82c-6c4c3c058bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:52.698850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f881a90-bf74-40b9-b82c-6c4c3c058bc2 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:52.701261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 13/22] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:52.708065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef6a9d-50fc-42c5-9f37-fb9b487a2d70 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:52.708799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef6a9d-50fc-42c5-9f37-fb9b487a2d70 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:52.708799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef6a9d-50fc-42c5-9f37-fb9b487a2d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:52.743611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e6ef6a9d-50fc-42c5-9f37-fb9b487a2d70 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:52.745249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e6ef6a9d-50fc-42c5-9f37-fb9b487a2d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:52.849800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173cd566-1ae0-437a-9650-bef28e20b865 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:52.850206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173cd566-1ae0-437a-9650-bef28e20b865 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:52.850602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173cd566-1ae0-437a-9650-bef28e20b865 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:52.852304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 14/23] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:06:52.868410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab72a390-7205-4932-b6e4-f7517f716fba None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:52.868717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab72a390-7205-4932-b6e4-f7517f716fba None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:52.869035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab72a390-7205-4932-b6e4-f7517f716fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:52.875033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 10/24] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:06:52.886313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f6d2c1-2fee-4cfc-b801-5aaa2008d10d None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:52.886712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f6d2c1-2fee-4cfc-b801-5aaa2008d10d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:52.887104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f6d2c1-2fee-4cfc-b801-5aaa2008d10d None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:52.889360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 11/25] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:06:52.908134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ab72a390-7205-4932-b6e4-f7517f716fba None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:52.910090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ab72a390-7205-4932-b6e4-f7517f716fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:52.919248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdd245-a64d-4129-9ff8-5125b7dc5c63 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:52.919652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdd245-a64d-4129-9ff8-5125b7dc5c63 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:52.920042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdd245-a64d-4129-9ff8-5125b7dc5c63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:52.943131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-7bbdd245-a64d-4129-9ff8-5125b7dc5c63 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:52.944545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7bbdd245-a64d-4129-9ff8-5125b7dc5c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:52.975303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:53.041622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 12/26] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:06:53.087804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 15/27] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:06:53.315145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbdd245-a64d-4129-9ff8-5125b7dc5c63 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:53.335409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876eba8b-d992-431e-b310-b0341aeb5da4 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:53.335882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876eba8b-d992-431e-b310-b0341aeb5da4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:53.336318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876eba8b-d992-431e-b310-b0341aeb5da4 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:53.338291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 16/28] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:06:53.361698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa549590-0fef-489a-a95f-4da77108b02d None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:53.362080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa549590-0fef-489a-a95f-4da77108b02d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:53.362294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa549590-0fef-489a-a95f-4da77108b02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:53.398845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-aa549590-0fef-489a-a95f-4da77108b02d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:53.405152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-aa549590-0fef-489a-a95f-4da77108b02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:53.583607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbdd245-a64d-4129-9ff8-5125b7dc5c63 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:53.589257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 17/29] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:06:53.591862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa549590-0fef-489a-a95f-4da77108b02d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:53.669156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:53.686699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:53.688398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:53.688798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:53.689075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:53.722381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:53.728629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:06:53.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:06:53.760076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:06:53.761510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:06:53.761510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:06:53.761510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:06:53.763700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 13/30] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:06:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 449 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:06:53.764958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:53.789710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:53.923261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa549590-0fef-489a-a95f-4da77108b02d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:53.958851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9190ac1a-1bd4-4008-a84b-be8527326340 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:54.015297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:54.016225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.016646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.017056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.017918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:54.019600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:54.027305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:54.028647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.029136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.029962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.031853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:54.034484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:54.196165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 18/31] 10.222.0.52 () {60 vars in 1131 bytes} [Sat Aug 27 02:06:53 2022] POST /identity/v3/roles => generated 219 bytes in 606 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:06:54.197398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:54.215627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 14/32] 10.222.0.52 () {60 vars in 1131 bytes} [Sat Aug 27 02:06:53 2022] POST /identity/v3/roles => generated 225 bytes in 451 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:06:54.217689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:54.305574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:54.341605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b04d01d-0fe0-43ce-8874-349591ddbbcd None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:54.361678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:54.408474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:54.409537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.409981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.410427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.411546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:06:54.412022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:54.414200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:54.446092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 19/33] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:06:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:54.452282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:54.466755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c820de53-fb0b-4709-b3f5-fc6374a75f08 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:54.563309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e52dc3-28f9-44b1-9d93-47e5810ef6ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:54.564242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e52dc3-28f9-44b1-9d93-47e5810ef6ba None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.564607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e52dc3-28f9-44b1-9d93-47e5810ef6ba None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.564877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e52dc3-28f9-44b1-9d93-47e5810ef6ba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.581797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e52dc3-28f9-44b1-9d93-47e5810ef6ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-f2e52dc3-28f9-44b1-9d93-47e5810ef6ba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:06:54.588587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:06:54.590031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 15/34] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:06:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 373 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:06:54.591233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa81112a-c4df-4f2f-9842-83535c3a428a None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.592176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa81112a-c4df-4f2f-9842-83535c3a428a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.592176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa81112a-c4df-4f2f-9842-83535c3a428a None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.593271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 16/35] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:54.597268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87364a4-2975-4af8-ba92-4015cd5d367b None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.597268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87364a4-2975-4af8-ba92-4015cd5d367b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.597268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87364a4-2975-4af8-ba92-4015cd5d367b None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.597268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 17/36] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:54.602969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87364a4-2975-4af8-ba92-4015cd5d367b None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:54.628530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c6aaa7-9400-4612-8df7-8ae9cb4cf639 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:54.692100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f996eaeb-9a3f-4089-9a3c-ce9e0f09a3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:54.692742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f996eaeb-9a3f-4089-9a3c-ce9e0f09a3ac None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.693025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f996eaeb-9a3f-4089-9a3c-ce9e0f09a3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.693391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f996eaeb-9a3f-4089-9a3c-ce9e0f09a3ac None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.694300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f996eaeb-9a3f-4089-9a3c-ce9e0f09a3ac None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:54.696455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f996eaeb-9a3f-4089-9a3c-ce9e0f09a3ac None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:54.723399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:54.740240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 20/37] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:06:54 2022] POST /identity/v3/projects => generated 301 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:06:54.745965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd5b02-ac0d-4b43-a356-52b5612e9958 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.746338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd5b02-ac0d-4b43-a356-52b5612e9958 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.746567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd5b02-ac0d-4b43-a356-52b5612e9958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.777795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5fcd5b02-ac0d-4b43-a356-52b5612e9958 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:54.779449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5fcd5b02-ac0d-4b43-a356-52b5612e9958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:54.808567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87364a4-2975-4af8-ba92-4015cd5d367b None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:54.864293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:54.877235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:54.878207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.878626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.879060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.879945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:06:54.880253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:54.882638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:54.898664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 21/38] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:06:54.899977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 18/39] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:06:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:54.906477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7246fce6-7611-4cee-839a-4bbf1b689334 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:54.907394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7246fce6-7611-4cee-839a-4bbf1b689334 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:54.909037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:54.910542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7246fce6-7611-4cee-839a-4bbf1b689334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:54.943775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7246fce6-7611-4cee-839a-4bbf1b689334 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:54.945283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7246fce6-7611-4cee-839a-4bbf1b689334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:55.077547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 22/40] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:06:55.082380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6c71f6-6f63-4e97-8936-61709ab37912 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:55.140588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:55.141467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:55.141756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:55.142063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] PATH_INFO: `/v3/users/6648709a4b6d470c80e9b42cb0d345ad` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:55.143991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] RBAC: Authorizing `identity:get_user(user_id=6648709a4b6d470c80e9b42cb0d345ad)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:55.145561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:55.148460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 19/41] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:06:54 2022] GET /identity/v3/users/6648709a4b6d470c80e9b42cb0d345ad => generated 247 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:55.153426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7246fce6-7611-4cee-839a-4bbf1b689334 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:55.239197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:55.247727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:55.309513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7246fce6-7611-4cee-839a-4bbf1b689334 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:55.322905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:55.354916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:55.355614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:55.356020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:55.356469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:55.357940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:55.359923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:55.362023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 23/42] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:06:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:55.368749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:55.391536 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bf1915-3fa2-4d03-8273-7ae790066a7f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:55.451611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:55.455288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:55.456301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:55.456301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:55.457295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:55.460622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:55.486518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 20/43] 10.222.0.52 () {60 vars in 1131 bytes} [Sat Aug 27 02:06:55 2022] POST /identity/v3/roles => generated 223 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:06:55.487765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:55.518988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a4e86d-cbea-40ae-ad95-9d2e33e4bfe7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:55.551489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:55.580861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:55.581821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:55.582269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:55.582777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] PATH_INFO: `/v3/domains/default/users/6648709a4b6d470c80e9b42cb0d345ad/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:55.585551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=6648709a4b6d470c80e9b42cb0d345ad, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:55.587312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:55.609228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 24/44] 10.222.0.52 () {58 vars in 1361 bytes} [Sat Aug 27 02:06:55 2022] PUT /identity/v3/domains/default/users/6648709a4b6d470c80e9b42cb0d345ad/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:06:55.704391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b338cc8-d7ee-481d-ae4d-4087f078f8c6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:55.747070 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:55.774775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9fbd5b-f4ad-47fe-9595-bb064c207090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:55.775449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9fbd5b-f4ad-47fe-9595-bb064c207090 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:55.775757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9fbd5b-f4ad-47fe-9595-bb064c207090 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:55.776033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9fbd5b-f4ad-47fe-9595-bb064c207090 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:55.778199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9fbd5b-f4ad-47fe-9595-bb064c207090 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:55.779633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9fbd5b-f4ad-47fe-9595-bb064c207090 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:55.781661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 21/45] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:06:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:55.788692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:55.937600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f2df3-4acb-49f5-9b76-5b9fd00bf0ee None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:55.978731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719cb66-7220-4892-958e-ae31657e4296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:55.979563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719cb66-7220-4892-958e-ae31657e4296 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:55.979651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719cb66-7220-4892-958e-ae31657e4296 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:55.979896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719cb66-7220-4892-958e-ae31657e4296 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:55.980585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7719cb66-7220-4892-958e-ae31657e4296 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:55.982098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7719cb66-7220-4892-958e-ae31657e4296 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:56.013825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 25/46] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:06:55 2022] POST /identity/v3/projects => generated 312 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:06:57.921513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea62852f-9f8e-4a65-be4b-14a1aa89489c None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:57.922318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea62852f-9f8e-4a65-be4b-14a1aa89489c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:57.922464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea62852f-9f8e-4a65-be4b-14a1aa89489c None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:57.924218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 22/47] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:57.931275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9903cec-3b26-47b9-9100-06d30a0e34b9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:57.931981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9903cec-3b26-47b9-9100-06d30a0e34b9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:57.932237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9903cec-3b26-47b9-9100-06d30a0e34b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:57.963301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-a9903cec-3b26-47b9-9100-06d30a0e34b9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:57.964527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a9903cec-3b26-47b9-9100-06d30a0e34b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:58.004779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:58.051546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0aaf4-843d-480a-a463-628104fa19cb None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:58.052168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0aaf4-843d-480a-a463-628104fa19cb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:58.052733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0aaf4-843d-480a-a463-628104fa19cb None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:58.055282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 23/48] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:06:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:06:58.062868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 26/49] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:06:58.064141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d6a661-f730-4f03-bfb1-bd77b5aa4c0e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:58.064688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d6a661-f730-4f03-bfb1-bd77b5aa4c0e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:58.065236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d6a661-f730-4f03-bfb1-bd77b5aa4c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:58.097762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-70d6a661-f730-4f03-bfb1-bd77b5aa4c0e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:06:58.100405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-70d6a661-f730-4f03-bfb1-bd77b5aa4c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:06:58.300767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 24/50] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:06:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:06:58.505024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9903cec-3b26-47b9-9100-06d30a0e34b9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:58.606599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d6a661-f730-4f03-bfb1-bd77b5aa4c0e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:58.662062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9903cec-3b26-47b9-9100-06d30a0e34b9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:58.701099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:58.701769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:58.702050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:58.702327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:58.719209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:06:58.724381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:06:58.726029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:06:58.727698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 27/51] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:06:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:06:58.733228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:58.741692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d6a661-f730-4f03-bfb1-bd77b5aa4c0e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:58.768737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:58.769147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:58.798116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:58.798765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:58.799046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:58.799391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:58.800569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:58.802149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:58.803825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 25/52] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:06:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:58.811657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:58.967052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fea1af-d060-4eee-a0e5-67844f774b00 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:59.019173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:59.019952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:59.020415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:59.020863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:59.021925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:59.023716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:59.041400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6907899-8ff3-4658-b804-03f76545f6c2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:59.058722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 26/53] 10.222.0.52 () {60 vars in 1140 bytes} [Sat Aug 27 02:06:58 2022] POST /identity/v3/projects => generated 298 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:06:59.101850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:59.102768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:59.103071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:59.103364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:59.104099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:06:59.104448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:59.105888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:59.119128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 28/54] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:06:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:59.127023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:59.196417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:59.309332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c0c5b1-eaef-4626-8717-55e84b092477 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:59.355522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:59.356235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:59.356520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:59.356793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] PATH_INFO: `/v3/users/6648709a4b6d470c80e9b42cb0d345ad` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:59.358012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] RBAC: Authorizing `identity:get_user(user_id=6648709a4b6d470c80e9b42cb0d345ad)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:59.359487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:59.361631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 27/55] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:06:59 2022] GET /identity/v3/users/6648709a4b6d470c80e9b42cb0d345ad => generated 247 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:59.368322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:59.498447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:59.524431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccb1ade-eace-45cd-9ec9-c0f4afe19596 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:59.564426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448a4c1-2f25-441e-a555-033efb4d880b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:59.565144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448a4c1-2f25-441e-a555-033efb4d880b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:59.565515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448a4c1-2f25-441e-a555-033efb4d880b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:59.565868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448a4c1-2f25-441e-a555-033efb4d880b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:59.567296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1448a4c1-2f25-441e-a555-033efb4d880b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:59.569335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1448a4c1-2f25-441e-a555-033efb4d880b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:59.571107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 29/56] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:06:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:06:59.577361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:06:59.737744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4cc805-4e8a-4e72-927c-c3b3e06f3100 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:06:59.746103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:06:59.788750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cd559e-ce9f-4afe-b2c5-36ac8db60227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:06:59.789805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cd559e-ce9f-4afe-b2c5-36ac8db60227 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:06:59.790288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cd559e-ce9f-4afe-b2c5-36ac8db60227 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:06:59.790756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cd559e-ce9f-4afe-b2c5-36ac8db60227 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:06:59.792122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cd559e-ce9f-4afe-b2c5-36ac8db60227 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.domain.id=default, user.id=6648709a4b6d470c80e9b42cb0d345ad) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:06:59.792703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cd559e-ce9f-4afe-b2c5-36ac8db60227 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:06:59.795395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cd559e-ce9f-4afe-b2c5-36ac8db60227 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:06:59.807670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 28/57] 10.222.0.52 () {56 vars in 1322 bytes} [Sat Aug 27 02:06:59 2022] GET /identity/v3/role_assignments?user.id=6648709a4b6d470c80e9b42cb0d345ad&scope.domain.id=default&role.id=b83e461b8f6643fba45f427d28498c41 => generated 530 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:01.244199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f4d8c-190a-42fd-8102-42eb289fe92f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:01.244771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f4d8c-190a-42fd-8102-42eb289fe92f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:01.244771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f4d8c-190a-42fd-8102-42eb289fe92f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:01.246141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 30/58] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:01.254634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaa566-2278-45f7-9fab-4843358f4efb None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:01.255383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaa566-2278-45f7-9fab-4843358f4efb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:01.255383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaa566-2278-45f7-9fab-4843358f4efb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:01.275707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2bfaa566-2278-45f7-9fab-4843358f4efb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:01.276930 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2bfaa566-2278-45f7-9fab-4843358f4efb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:01.357366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 29/59] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:01.628013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f4d8c-190a-42fd-8102-42eb289fe92f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:01.754512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f4d8c-190a-42fd-8102-42eb289fe92f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:01.804610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a6bd21-a468-4f3e-96b5-0db67043cc4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:01.805621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6bd21-a468-4f3e-96b5-0db67043cc4b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:01.806077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6bd21-a468-4f3e-96b5-0db67043cc4b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:01.806519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6bd21-a468-4f3e-96b5-0db67043cc4b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:01.808184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a6bd21-a468-4f3e-96b5-0db67043cc4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:01.809696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a6bd21-a468-4f3e-96b5-0db67043cc4b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:01.811315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 31/60] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:01.817937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaa566-2278-45f7-9fab-4843358f4efb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:01.954703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:01.959010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaa566-2278-45f7-9fab-4843358f4efb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:02.008599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da80111-300a-468d-b0a4-9881c319020a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:02.009321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da80111-300a-468d-b0a4-9881c319020a None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:02.009617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da80111-300a-468d-b0a4-9881c319020a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:02.009920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da80111-300a-468d-b0a4-9881c319020a None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:02.011067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da80111-300a-468d-b0a4-9881c319020a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:02.013760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da80111-300a-468d-b0a4-9881c319020a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:02.051888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 30/61] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:07:01 2022] POST /identity/v3/users => generated 275 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:06.441148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4984b58b-a807-43c6-99e9-02060a1b748d None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:06.442542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4984b58b-a807-43c6-99e9-02060a1b748d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:06.443235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4984b58b-a807-43c6-99e9-02060a1b748d None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:06.446097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 32/62] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:06 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:06.452976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73d95f6-ab0f-4ccf-a608-4a20d2f2719c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:06.454182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73d95f6-ab0f-4ccf-a608-4a20d2f2719c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:06.454182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73d95f6-ab0f-4ccf-a608-4a20d2f2719c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:06.494351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d73d95f6-ab0f-4ccf-a608-4a20d2f2719c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:06.495692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d73d95f6-ab0f-4ccf-a608-4a20d2f2719c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:06.615507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 31/63] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:06.878444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea48ad8d-2e32-451a-8c8c-4743fe31f7ad None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:06.879313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea48ad8d-2e32-451a-8c8c-4743fe31f7ad None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:06.879952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea48ad8d-2e32-451a-8c8c-4743fe31f7ad None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:06.881996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 33/64] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:06.900813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc5387-f385-4bab-8d7c-4ae83c450e73 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:06.901210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc5387-f385-4bab-8d7c-4ae83c450e73 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:06.901585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc5387-f385-4bab-8d7c-4ae83c450e73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:06.902724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee3f1c8-832b-48aa-8ac3-b9a0afb0dc75 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:06.903335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee3f1c8-832b-48aa-8ac3-b9a0afb0dc75 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:06.903922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee3f1c8-832b-48aa-8ac3-b9a0afb0dc75 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:06.906514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 34/65] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:06.918375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebc4040-07de-4734-85a6-3f43134510b6 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:06.919249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebc4040-07de-4734-85a6-3f43134510b6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:06.920007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebc4040-07de-4734-85a6-3f43134510b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:06.956532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e4bc5387-f385-4bab-8d7c-4ae83c450e73 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:06.960505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e4bc5387-f385-4bab-8d7c-4ae83c450e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:06.971520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-aebc4040-07de-4734-85a6-3f43134510b6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:06.981410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-aebc4040-07de-4734-85a6-3f43134510b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:07.042117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:07.144923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 32/66] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:07.147712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e019a1c-ad07-4f94-830a-517eb02680c4 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:07.148175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 35/67] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:07.148623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e019a1c-ad07-4f94-830a-517eb02680c4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:07.149252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e019a1c-ad07-4f94-830a-517eb02680c4 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:07.150269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebc4040-07de-4734-85a6-3f43134510b6 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:07.151893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 33/68] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:07 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:07.167799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4e4114-4add-4a5f-9e59-fe42adcee3f2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:07.168602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4e4114-4add-4a5f-9e59-fe42adcee3f2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:07.169231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4e4114-4add-4a5f-9e59-fe42adcee3f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:07.214703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-7d4e4114-4add-4a5f-9e59-fe42adcee3f2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:07.215989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7d4e4114-4add-4a5f-9e59-fe42adcee3f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:07.245555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:07.381298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 34/69] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:07.382324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebc4040-07de-4734-85a6-3f43134510b6 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:07.383649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc6b251-a9e8-4c8a-8a5a-2b4ba75a61ea None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:07.384304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc6b251-a9e8-4c8a-8a5a-2b4ba75a61ea None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:07.385191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc6b251-a9e8-4c8a-8a5a-2b4ba75a61ea None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:07.387446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 35/70] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:07.395600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dbbdc8-02d0-4d1c-8498-d3efc94ad1d5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:07.396263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dbbdc8-02d0-4d1c-8498-d3efc94ad1d5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:07.396933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dbbdc8-02d0-4d1c-8498-d3efc94ad1d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:07.446922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e8dbbdc8-02d0-4d1c-8498-d3efc94ad1d5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:07.448657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e8dbbdc8-02d0-4d1c-8498-d3efc94ad1d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:07.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:07.457792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:07.458266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:07.458614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:07.496362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:07.508335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:07.509467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:07.510524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:07.511984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 36/71] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 363 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:07.525403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:07.549577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:07.664715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 36/72] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:07.666429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dbbdc8-02d0-4d1c-8498-d3efc94ad1d5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:07.699613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2610791-cc55-41a3-a866-cd5ce88c0dbb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:07.781824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:07.781824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:07.781824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:07.781824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:07.782572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:07.782572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:07.782572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:07.794777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 37/73] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:07.797872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:07.836043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:07.881415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dbbdc8-02d0-4d1c-8498-d3efc94ad1d5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:07.948608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:07.949718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:07.950266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:07.950762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:07.954283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:07.968565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:07.976727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:07.977999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:07.979060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:07.985572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 37/74] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:07.987058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:07.996711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e0f7d-8927-4819-b811-aacb5718afe1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.055801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.056392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.056767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.057149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.072347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:08.077937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:08.080518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:08.081797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 38/75] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:07:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:08.082878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:08.160818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e6fbff-8c5b-47b3-93af-8501b5ad1c85 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.202513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.203664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.204065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.204479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.208265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:08.225794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:08.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:08.237870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 38/76] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:07 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:08.237870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:08.251078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deb4200-ef5c-468a-8314-100a9c3d1346 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.301633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.302280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.302564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.302873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.306672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:08.328511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:08.339192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:08.340271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 39/77] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:07:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:08.347183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:08.394942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9eefbc-53d4-4618-8a54-e446eec5200b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.443932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.444883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.445355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.445793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.446893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:08.447291 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:08.449462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:08.470822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 39/78] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:08.470822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:08.526731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194f5daa-75ce-4788-90f7-7438045de667 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.560531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:08.582988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.583978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.584415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.584839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.586955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:08.589481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:08.592178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 40/79] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:08.593433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:08.626845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657fbff4-ae9a-4c12-b0d7-e3efa76abb60 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.676330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.678905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.678905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.678905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.678905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:08.678905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:08.680412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:08.692388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 40/80] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:08.693305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:08.699866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:08.758498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc86ac-7613-4467-994b-4fce19327787 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.808344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.808971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.809272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.809544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.846299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e561ff1c-3d48-41aa-9f9f-02c9698f7a77 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:08.868182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:08.896481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:08.897215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:08.897533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:08.897908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:08.898394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:08.898744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:08.900085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:08.907867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:08.914519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 41/81] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:07:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:08.914742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:08.914842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:09.022145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:09.073019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 41/82] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:09.073710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:09.079825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37562c0-ab4a-4030-9922-6f01c9737d50 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:09.125806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:09.126740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:09.127109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:09.127416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:09.129201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:09.130726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:09.132411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 42/83] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:08 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:09.138773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:09.207139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:09.221658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58c8e34-fd92-43f9-acc2-10a3653a18f2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:09.276547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:09.277069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:09.277389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:09.277667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:09.278394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:09.280036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:09.291525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966ad57b-f203-4459-af91-b8a36d3801e8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:09.313963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 42/84] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:07:09 2022] POST /identity/v3/users => generated 289 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:09.314692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:09.349305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:09.355024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:09.356617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:09.357438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:09.357799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] PATH_INFO: `/v3/users/6648709a4b6d470c80e9b42cb0d345ad` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:09.360233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] RBAC: Authorizing `identity:get_user(user_id=6648709a4b6d470c80e9b42cb0d345ad)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:09.363114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:09.366276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 43/85] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:09 2022] GET /identity/v3/users/6648709a4b6d470c80e9b42cb0d345ad => generated 247 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:09.373913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:09.507240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47feab79-205d-4c69-9114-53ceab597fd5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:09.562901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903b2996-5140-4e48-b899-aea7d390678b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:09.591199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:09.603499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:09.604511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:09.604961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:09.605764 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:09.608104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:09.610731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:09.614517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 43/86] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:09 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:09.616310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:09.631051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:09.632055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:09.632524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:09.632927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:09.635693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:09.638904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:09.642344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 44/87] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:09 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:09.643395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:09.799519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:09.829997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e27c12e-85d7-43af-99ff-fff4f9e9242b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:09.846427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8880c64-023d-439d-b19f-a88e75f048fb None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:09.896178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:09.898301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:09.899694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:09.900442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:09.929110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:09.930176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:09.930662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:09.931120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:10.042784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:10.059103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:10.063225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:10.080561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:10.083654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:10.216374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:10.244522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 44/88] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 629 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:10.245698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:10.246531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 45/89] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 604 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:10.253418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:10.373679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:10.391235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458406b4-2df9-49c9-bd13-900616f94360 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:10.392136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609ede4f-ba62-40d1-a7bd-2b3ab962aaf2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:10.434827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:10.435945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:10.436605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:10.437081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:10.438422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:10.438944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:10.440621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:10.447805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 46/90] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:10 2022] GET /identity/v3/role_assignments?user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 259 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:10.449003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:10.455187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:10.456210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:10.456704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:10.457214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:10.570016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2938a72-ad90-4094-b9d3-752e4fc65f32 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:10.628834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:10.654381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:10.657530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:10.658661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:10.663889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] PATH_INFO: `/v3/projects/8f8d1152ed1e402fade470adfdc1f70f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:10.666665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] RBAC: Authorizing `identity:get_project(project_id=8f8d1152ed1e402fade470adfdc1f70f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:10.669440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:10.672849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 47/91] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:10 2022] GET /identity/v3/projects/8f8d1152ed1e402fade470adfdc1f70f => generated 312 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:10.674851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:10.684740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:10.687601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:10.717870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:10.872479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 45/92] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 627 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:10.873811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:10.906193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa64d6-87fc-43bf-9e11-7d3843a30a61 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:11.032008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:11.033065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:11.033545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:11.033981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:11.035867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:11.038554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:11.041720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 48/93] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:10 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:11.042800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:11.169440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:11.216814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:11.226332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b16d6-f933-4877-960b-0d9fb3210ffa None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:11.247683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b46fd-f60d-422e-ad48-008d6eabbb3d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:11.313152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:11.314552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:11.315247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:11.315889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:11.318080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:11.321226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:11.324650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 49/94] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:11 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:11.326276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:11.347657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:11.348946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:11.349236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:11.349695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:11.350995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=8f8d1152ed1e402fade470adfdc1f70f, user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:11.351581 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:11.358968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:11.390941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 46/95] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:10 2022] GET /identity/v3/role_assignments?user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7&scope.project.id=8f8d1152ed1e402fade470adfdc1f70f&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 259 bytes in 518 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:11.397152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:11.546006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17993c21-3d26-4939-bea2-a598637446aa None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:11.594134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51efcd95-832e-480d-8536-5c0505f385b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:11.595290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51efcd95-832e-480d-8536-5c0505f385b5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:11.597024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51efcd95-832e-480d-8536-5c0505f385b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:11.598227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51efcd95-832e-480d-8536-5c0505f385b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:11.599180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51efcd95-832e-480d-8536-5c0505f385b5 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=6648709a4b6d470c80e9b42cb0d345ad) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:11.600002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51efcd95-832e-480d-8536-5c0505f385b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:11.603623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51efcd95-832e-480d-8536-5c0505f385b5 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:11.618035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 50/96] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:11 2022] GET /identity/v3/role_assignments?user.id=6648709a4b6d470c80e9b42cb0d345ad&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:11.695308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14dbaf9-fae5-4ebc-ae45-df8e0b694eb9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:11.779264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf63cda5-1404-4207-af8c-6e02161817f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:11.779997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf63cda5-1404-4207-af8c-6e02161817f0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:11.780278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf63cda5-1404-4207-af8c-6e02161817f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:11.780545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf63cda5-1404-4207-af8c-6e02161817f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:11.781385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf63cda5-1404-4207-af8c-6e02161817f0 None admin] RBAC: Adding query filter params (role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:11.781690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf63cda5-1404-4207-af8c-6e02161817f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:11.783203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf63cda5-1404-4207-af8c-6e02161817f0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:11.790609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 47/97] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:11 2022] GET /identity/v3/role_assignments?user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 259 bytes in 399 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:12.517878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd1e493-c847-461b-8905-b33dbdc59801 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:12.518899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd1e493-c847-461b-8905-b33dbdc59801 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:12.519381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd1e493-c847-461b-8905-b33dbdc59801 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:12.522025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 51/98] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:12.528508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848b0e2-8394-4feb-a59c-8af75ee37f79 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:12.529193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848b0e2-8394-4feb-a59c-8af75ee37f79 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:12.529708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848b0e2-8394-4feb-a59c-8af75ee37f79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:12.566559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-a848b0e2-8394-4feb-a59c-8af75ee37f79 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:12.568503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a848b0e2-8394-4feb-a59c-8af75ee37f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:12.587849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:12.675523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630f127-684d-43fa-9fc4-49f5e481f9f6 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:12.675983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630f127-684d-43fa-9fc4-49f5e481f9f6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:12.676533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630f127-684d-43fa-9fc4-49f5e481f9f6 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:12.678466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 52/99] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:12.685263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aec06a-729a-482b-ac83-87b4910fda23 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:12.685632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aec06a-729a-482b-ac83-87b4910fda23 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:12.686326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aec06a-729a-482b-ac83-87b4910fda23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:12.713819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-09aec06a-729a-482b-ac83-87b4910fda23 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:12.715125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-09aec06a-729a-482b-ac83-87b4910fda23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:12.733780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:12.750417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 48/100] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:12.861073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 53/101] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:13.461340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848b0e2-8394-4feb-a59c-8af75ee37f79 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:13.488495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aec06a-729a-482b-ac83-87b4910fda23 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:13.677841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848b0e2-8394-4feb-a59c-8af75ee37f79 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:13.767180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aec06a-729a-482b-ac83-87b4910fda23 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:13.784631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:13.785392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:13.785674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:13.785885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:13.817339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:13.835128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:13.836723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:13.838652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 49/102] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 377 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:13.849332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:13.849733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:13.850064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:13.850349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:13.850621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:13.858403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:13.861032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:13.862540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:13.863892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 54/103] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 375 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:13.875900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:13.919349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:14.142801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89429b21-bc1e-4717-9f5b-3480b6c51af5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:14.146842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2779deb-e1db-4575-a311-f5ad5701f93c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:14.209056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced6df3b-6156-4681-a01a-3f2239ad238c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:14.209722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced6df3b-6156-4681-a01a-3f2239ad238c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:14.210089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced6df3b-6156-4681-a01a-3f2239ad238c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:14.210363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced6df3b-6156-4681-a01a-3f2239ad238c None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:14.211268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced6df3b-6156-4681-a01a-3f2239ad238c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:14.211774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced6df3b-6156-4681-a01a-3f2239ad238c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:14.213357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced6df3b-6156-4681-a01a-3f2239ad238c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:14.226090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 50/104] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 378 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:14.228690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e5f78b-56b2-41ec-96d0-bbf175dab0ab None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:14.229170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e5f78b-56b2-41ec-96d0-bbf175dab0ab None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:14.229653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e5f78b-56b2-41ec-96d0-bbf175dab0ab None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:14.232228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 51/105] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:14.239078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0848d2c0-6d45-47d1-964f-61ebb8882ee5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:14.239534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0848d2c0-6d45-47d1-964f-61ebb8882ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:14.239998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0848d2c0-6d45-47d1-964f-61ebb8882ee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:14.259971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:14.260616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:14.260901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:14.261201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:14.262285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:14.262779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-0848d2c0-6d45-47d1-964f-61ebb8882ee5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:14.264044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:14.264309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0848d2c0-6d45-47d1-964f-61ebb8882ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:14.319524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 55/106] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:07:13 2022] POST /identity/v3/projects => generated 302 bytes in 445 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:14.320562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:14.456807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:14.481951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 52/107] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:14.545825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:14.623772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96ec3d2-e71b-4c3a-9dc2-51318b201191 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:14.700439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039d37b-91dd-4323-a5ca-010a06ba260a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:14.701953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039d37b-91dd-4323-a5ca-010a06ba260a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:14.702401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039d37b-91dd-4323-a5ca-010a06ba260a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:14.702866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039d37b-91dd-4323-a5ca-010a06ba260a None admin] PATH_INFO: `/v3/users/efb9f072219f40b896cd2c251834dafe` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:14.733686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039d37b-91dd-4323-a5ca-010a06ba260a None admin] RBAC: Authorizing `identity:get_user(user_id=efb9f072219f40b896cd2c251834dafe)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:14.735818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039d37b-91dd-4323-a5ca-010a06ba260a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:14.737862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 56/108] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:14 2022] GET /identity/v3/users/efb9f072219f40b896cd2c251834dafe => generated 289 bytes in 418 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:14.747242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0848d2c0-6d45-47d1-964f-61ebb8882ee5 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:14.921405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0848d2c0-6d45-47d1-964f-61ebb8882ee5 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:14.996924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:14.997673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:14.997962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:14.998239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:15.105405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:15.118841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:15.121511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:15.191520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2030a390-d29a-4e4b-a28a-79f1c9d6244c None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:15.192167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2030a390-d29a-4e4b-a28a-79f1c9d6244c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:15.192411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2030a390-d29a-4e4b-a28a-79f1c9d6244c None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:15.193758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 57/109] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:15.199635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973377ee-6ac5-48c6-b262-66f038e7b1ed None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:15.199986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973377ee-6ac5-48c6-b262-66f038e7b1ed None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:15.200193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973377ee-6ac5-48c6-b262-66f038e7b1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:15.239344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-973377ee-6ac5-48c6-b262-66f038e7b1ed None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:15.240589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-973377ee-6ac5-48c6-b262-66f038e7b1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:15.285367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 53/110] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 539 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:15.289753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:15.367991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 58/111] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:15.369699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3931fa7e-8d6e-44f4-96ec-72cba43f5d08 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:15.369973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3931fa7e-8d6e-44f4-96ec-72cba43f5d08 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:15.370262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3931fa7e-8d6e-44f4-96ec-72cba43f5d08 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:15.373303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 59/112] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:15.373303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3931fa7e-8d6e-44f4-96ec-72cba43f5d08 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:15.492072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:15.496258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:15.508823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffee8f80-b5dc-4465-bdf8-ce48d2d6c9d4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:15.575139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7494b80-1509-4e95-9028-3d77789ec6b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:15.576985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7494b80-1509-4e95-9028-3d77789ec6b5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:15.577791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7494b80-1509-4e95-9028-3d77789ec6b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:15.578490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7494b80-1509-4e95-9028-3d77789ec6b5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:15.601515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7494b80-1509-4e95-9028-3d77789ec6b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-c7494b80-1509-4e95-9028-3d77789ec6b5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:15.613503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:15.615185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:15.616666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:15.621526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 54/113] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:15.624599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e375af-b244-4f47-8efd-764b2e2fa888 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:15.625470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e375af-b244-4f47-8efd-764b2e2fa888 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:15.626202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e375af-b244-4f47-8efd-764b2e2fa888 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:15.658418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-a4e375af-b244-4f47-8efd-764b2e2fa888 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:15.661021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a4e375af-b244-4f47-8efd-764b2e2fa888 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:15.686004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3931fa7e-8d6e-44f4-96ec-72cba43f5d08 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:15.785759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:15.786543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:15.786887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:15.787181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:15.790210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:15.792144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:15.794252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 60/114] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:15 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 422 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:15.795001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:15.814843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 55/115] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:15.816617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e375af-b244-4f47-8efd-764b2e2fa888 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:15.961923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:15.981909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764abd4a-add7-411c-9e43-a36d8b7ac0c9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:15.988784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e375af-b244-4f47-8efd-764b2e2fa888 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:16.019123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:16.042503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:16.043768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.044533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.045291 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.046938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:16.047789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:16.050273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:16.053993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbc33cc-244c-4b24-9036-9bed4669f109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:16.055104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc33cc-244c-4b24-9036-9bed4669f109 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.055630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc33cc-244c-4b24-9036-9bed4669f109 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.056135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc33cc-244c-4b24-9036-9bed4669f109 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.062604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 61/116] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:16.063892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:16.076069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbc33cc-244c-4b24-9036-9bed4669f109 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-ddbc33cc-244c-4b24-9036-9bed4669f109 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:16.097397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:16.099478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 56/117] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:07:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:16.104276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5235989a-0e63-4c4b-86ff-d4b90470011e None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.107561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5235989a-0e63-4c4b-86ff-d4b90470011e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.107561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5235989a-0e63-4c4b-86ff-d4b90470011e None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.107561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 57/118] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:16.114307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5235989a-0e63-4c4b-86ff-d4b90470011e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:16.252677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3afe9cb-4725-41f1-9384-e9cea7819fe7 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:16.293315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5235989a-0e63-4c4b-86ff-d4b90470011e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:16.317585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:16.318482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.318901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.319361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.320900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] RBAC: Adding query filter params (role.id=b19c6f50e2f542aa81ec4cf175cba448, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=efb9f072219f40b896cd2c251834dafe) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:16.321442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:16.323798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:16.345159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 62/119] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:16 2022] GET /identity/v3/role_assignments?user.id=efb9f072219f40b896cd2c251834dafe&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=b19c6f50e2f542aa81ec4cf175cba448 => generated 259 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:16.345159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:16.350281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7f0700-c73a-4135-aece-fc22cb1345d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:16.352903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7f0700-c73a-4135-aece-fc22cb1345d7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.353597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7f0700-c73a-4135-aece-fc22cb1345d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.354161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7f0700-c73a-4135-aece-fc22cb1345d7 None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.356565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7f0700-c73a-4135-aece-fc22cb1345d7 None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:16.358363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7f0700-c73a-4135-aece-fc22cb1345d7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:16.361784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 58/120] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:16 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:16.364581 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb442c60-704a-43ce-9926-e08c34f1675a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.364943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb442c60-704a-43ce-9926-e08c34f1675a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.365638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb442c60-704a-43ce-9926-e08c34f1675a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.396838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-cb442c60-704a-43ce-9926-e08c34f1675a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:16.398356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-cb442c60-704a-43ce-9926-e08c34f1675a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:16.419781 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:16.425427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:16.504760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4f17f3-93d1-478d-a966-dab09ad9af04 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:16.520030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 59/121] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:16.521342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb442c60-704a-43ce-9926-e08c34f1675a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:16.565921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:16.566555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.566830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.567105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.567793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:16.568094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:16.572067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:16.583701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 63/122] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:16.590603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:16.745544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb442c60-704a-43ce-9926-e08c34f1675a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:16.820464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa664e2b-3258-478d-8a83-7b29078c39d4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:16.836304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:16.837251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.837610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.838012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.849789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:16.855452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:16.864162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:16.865898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:16.867380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:16.870913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 60/123] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 350 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:16.878878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:16.885813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:16.924192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:16.925109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:16.925567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:16.925987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:16.928862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:16.930528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:16.933366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 64/124] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:16 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:16.939425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.085066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6611366-7916-47ad-bae6-5c354604c6a0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:17.121839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fabb43-c0e2-455f-a600-4e178eb75335 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:17.134403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:17.135471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:17.136060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:17.136798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] PATH_INFO: `/v3/users/6648709a4b6d470c80e9b42cb0d345ad` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:17.139174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] RBAC: Authorizing `identity:get_user(user_id=6648709a4b6d470c80e9b42cb0d345ad)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:17.142137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:17.145851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 61/125] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:16 2022] GET /identity/v3/users/6648709a4b6d470c80e9b42cb0d345ad => generated 247 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:17.147531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.184487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:17.185530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:17.185964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:17.186524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:17.187639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:17.188121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:17.190663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:17.213157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 65/126] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:17.213157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.270595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:17.281611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:17.315064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ee0540-44fc-4f18-9004-1f6d1e64359b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:17.370559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:17.371778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:17.372252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:17.372881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:17.376447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] RBAC: Authorizing `identity:create_grant(project_id=abd8cdfe3c4b488eb590b64706ada806, user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:17.378740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:17.391082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbecdc7-c4ac-457c-8092-833df8f749e8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:17.405090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 62/127] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:17 2022] PUT /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:17.406193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.461843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:17.463399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:17.464084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:17.464940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:17.485506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:17.496993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:17.498287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:17.499398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:17.503101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 66/128] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:07:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:17.504340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.594739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb638fe6-3396-4740-bb01-14c44b148583 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:17.647111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:17.657801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:17.664830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:17.666504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:17.667157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:17.669951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:17.673065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:17.676817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 63/129] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:17 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:17.678460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.687026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:17.692337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c943c9-063c-4b6c-bd9d-ebd3e475063e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:17.752189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:17.753811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:17.754589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:17.755296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:17.757918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:17.760816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:17.764062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 67/130] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:17 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:17.765924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.856888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1471012d-ffc9-4979-9369-bd2387fa4cd9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:17.930131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:17.930131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:17.930464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:17.930668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:17.932112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:17.932769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:17.935212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:17.950866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 64/131] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:07:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:17.952335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:17.966918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9378924c-8b37-4f8a-b059-ab9b3ff8e9ba None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:18.022725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:18.024538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:18.025420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:18.026163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/6648709a4b6d470c80e9b42cb0d345ad/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:18.030392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] RBAC: Authorizing `identity:create_grant(project_id=abd8cdfe3c4b488eb590b64706ada806, user_id=6648709a4b6d470c80e9b42cb0d345ad, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:18.032957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:18.053475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:18.055991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 68/132] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:17 2022] PUT /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/6648709a4b6d470c80e9b42cb0d345ad/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:18.057312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:18.104935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:18.125940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a9927-eefe-4755-90bb-b86b3506c8f3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:18.175030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c32cd12-26bb-46d3-b2f9-635574e804a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:18.176469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c32cd12-26bb-46d3-b2f9-635574e804a8 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:18.177328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c32cd12-26bb-46d3-b2f9-635574e804a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:18.178125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c32cd12-26bb-46d3-b2f9-635574e804a8 None admin] PATH_INFO: `/v3/projects/8f8d1152ed1e402fade470adfdc1f70f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:18.180978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c32cd12-26bb-46d3-b2f9-635574e804a8 None admin] RBAC: Authorizing `identity:get_project(project_id=8f8d1152ed1e402fade470adfdc1f70f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:18.183952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c32cd12-26bb-46d3-b2f9-635574e804a8 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:18.186933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 65/133] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:17 2022] GET /identity/v3/projects/8f8d1152ed1e402fade470adfdc1f70f => generated 312 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:18.193165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c062cd92-0bdc-44a6-8b51-ffb07b3a6fcc None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:18.193904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c062cd92-0bdc-44a6-8b51-ffb07b3a6fcc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:18.194515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c062cd92-0bdc-44a6-8b51-ffb07b3a6fcc None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:18.196247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 66/134] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:18.197648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c062cd92-0bdc-44a6-8b51-ffb07b3a6fcc None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:18.254516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813c471e-b966-4869-a8b7-4365b89e34f1 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:18.320887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f90702-cca8-47de-ae88-3c379d24a847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:18.321717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f90702-cca8-47de-ae88-3c379d24a847 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:18.321717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f90702-cca8-47de-ae88-3c379d24a847 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:18.321879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f90702-cca8-47de-ae88-3c379d24a847 None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:18.323849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f90702-cca8-47de-ae88-3c379d24a847 None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:18.327204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f90702-cca8-47de-ae88-3c379d24a847 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:18.329877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 69/135] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:18 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:18.331253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945ed9c0-273b-4005-80f7-ca2174277522 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:18.331540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945ed9c0-273b-4005-80f7-ca2174277522 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:18.331813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945ed9c0-273b-4005-80f7-ca2174277522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:18.364930 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-945ed9c0-273b-4005-80f7-ca2174277522 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:18.367227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-945ed9c0-273b-4005-80f7-ca2174277522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:18.370229 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c062cd92-0bdc-44a6-8b51-ffb07b3a6fcc None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:18.428442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:18.442810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:18.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:18.444200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:18.444569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] PATH_INFO: `/v3/projects/8f8d1152ed1e402fade470adfdc1f70f/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:18.450373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] RBAC: Authorizing `identity:create_grant(project_id=8f8d1152ed1e402fade470adfdc1f70f, user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:18.454375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:18.477995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 67/136] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:18 2022] PUT /identity/v3/projects/8f8d1152ed1e402fade470adfdc1f70f/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:07:18.479939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:18.481508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 70/137] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:18.652330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d65cda7-f02a-4805-b164-aa1019686114 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:18.716478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:18.717467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:18.717931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:18.718347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:18.720333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:18.723127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:18.726236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 68/138] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:18 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:18.736011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945ed9c0-273b-4005-80f7-ca2174277522 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:18.786830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:18.867593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:18.956735 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945ed9c0-273b-4005-80f7-ca2174277522 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:18.984508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:19.060858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8b2be6-0a7e-45d2-927e-07f94db5b9aa None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:19.078149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5432a3-227c-4506-86b4-c3513b52f385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:19.079233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5432a3-227c-4506-86b4-c3513b52f385 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:19.079678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5432a3-227c-4506-86b4-c3513b52f385 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:19.080143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5432a3-227c-4506-86b4-c3513b52f385 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:19.083959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5432a3-227c-4506-86b4-c3513b52f385 None admin] RBAC: Authorizing `identity:create_grant(project_id=abd8cdfe3c4b488eb590b64706ada806, user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7, role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:19.087440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5432a3-227c-4506-86b4-c3513b52f385 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:19.110677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 71/139] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:18 2022] PUT /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:19.115383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba3d5fa-5798-4c7d-b61b-3fc7fffff22d None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:19.115383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba3d5fa-5798-4c7d-b61b-3fc7fffff22d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:19.115383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba3d5fa-5798-4c7d-b61b-3fc7fffff22d None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:19.116415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 72/140] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:19.124451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17033e7-b6ac-471f-a1b1-95851a09c6eb None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:19.125043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17033e7-b6ac-471f-a1b1-95851a09c6eb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:19.125624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17033e7-b6ac-471f-a1b1-95851a09c6eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:19.161864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f17033e7-b6ac-471f-a1b1-95851a09c6eb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:19.164425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f17033e7-b6ac-471f-a1b1-95851a09c6eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:19.165838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:19.166928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:19.167447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:19.168044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:19.174486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:19.177094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:19.187385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 69/141] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:19.189649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:19.304058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 73/142] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:19.439387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f50ed-d942-4961-b054-b6b511365262 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:19.486001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:19.499318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:19.500028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:19.500305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:19.500574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:19.501298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:19.503775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:19.543542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 70/143] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:07:19 2022] POST /identity/v3/users => generated 283 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:07:19.976454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17033e7-b6ac-471f-a1b1-95851a09c6eb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:20.009170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:20.174722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17033e7-b6ac-471f-a1b1-95851a09c6eb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:20.252668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:20.254283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:20.255120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:20.255821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:20.273957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:20.283836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:20.285564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:20.287071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:20.290944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 74/144] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:07:20.300210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:20.446122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59934044-6c07-4d61-9eea-c9f44f3594be None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:20.511056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:20.512570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:20.513050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:20.513533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:20.514599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:20.515281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:20.517025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:20.530395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 71/145] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:20.538827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:20.662586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:20.703986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ab4cb7-0dd4-4592-ac3d-d9cb6de8852b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:20.759303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:20.760190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:20.760560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:20.760940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] PATH_INFO: `/v3/users/efb9f072219f40b896cd2c251834dafe` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:20.762900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] RBAC: Authorizing `identity:get_user(user_id=efb9f072219f40b896cd2c251834dafe)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:20.764878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:20.767206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 75/146] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:20 2022] GET /identity/v3/users/efb9f072219f40b896cd2c251834dafe => generated 289 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:20.775552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:20.851149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:20.933430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293de999-7c08-49d5-91bc-7ace7bb8c314 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:20.987468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8604735-24ce-4ecd-ab6f-2162f4ebcefc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:20.988127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8604735-24ce-4ecd-ab6f-2162f4ebcefc None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:20.988410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8604735-24ce-4ecd-ab6f-2162f4ebcefc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:20.988659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8604735-24ce-4ecd-ab6f-2162f4ebcefc None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:20.990095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8604735-24ce-4ecd-ab6f-2162f4ebcefc None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:20.991865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8604735-24ce-4ecd-ab6f-2162f4ebcefc None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:20.993687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 72/147] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:20 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:21.000908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:21.168764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effff66c-2a61-4c6b-8b13-67823906f727 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:21.219493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f80083a-021c-4110-924b-489dd749c07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:21.220316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f80083a-021c-4110-924b-489dd749c07e None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:21.220613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f80083a-021c-4110-924b-489dd749c07e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:21.220908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f80083a-021c-4110-924b-489dd749c07e None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/efb9f072219f40b896cd2c251834dafe/roles/b19c6f50e2f542aa81ec4cf175cba448` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:21.223370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f80083a-021c-4110-924b-489dd749c07e None admin] RBAC: Authorizing `identity:create_grant(project_id=abd8cdfe3c4b488eb590b64706ada806, user_id=efb9f072219f40b896cd2c251834dafe, role_id=b19c6f50e2f542aa81ec4cf175cba448)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:21.225886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f80083a-021c-4110-924b-489dd749c07e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:21.245599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 76/148] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:20 2022] PUT /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806/users/efb9f072219f40b896cd2c251834dafe/roles/b19c6f50e2f542aa81ec4cf175cba448 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:22.258650 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92046b1-cb9f-49bd-a289-71149bf98efe None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:22.259522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92046b1-cb9f-49bd-a289-71149bf98efe None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:22.259522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92046b1-cb9f-49bd-a289-71149bf98efe None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:22.262083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 73/149] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:22.270393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2b7c2-e9ff-4da6-b5ca-643f77385809 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:22.271330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2b7c2-e9ff-4da6-b5ca-643f77385809 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:22.271330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2b7c2-e9ff-4da6-b5ca-643f77385809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:22.283233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:22.316437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2fe2b7c2-e9ff-4da6-b5ca-643f77385809 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:22.318773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2fe2b7c2-e9ff-4da6-b5ca-643f77385809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:22.425015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 77/150] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:22.540245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faa790d-c9e3-468c-a5a4-1677d621559b None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:22.540653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faa790d-c9e3-468c-a5a4-1677d621559b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:22.541057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faa790d-c9e3-468c-a5a4-1677d621559b None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:22.544001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 74/151] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:22.550614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828e892-cdb6-4f92-8a90-4fa0914c0641 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:22.550889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828e892-cdb6-4f92-8a90-4fa0914c0641 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:22.551190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828e892-cdb6-4f92-8a90-4fa0914c0641 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:22.576067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e828e892-cdb6-4f92-8a90-4fa0914c0641 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:22.577314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e828e892-cdb6-4f92-8a90-4fa0914c0641 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:22.697069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 78/152] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:23.260241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faa790d-c9e3-468c-a5a4-1677d621559b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:23.455017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faa790d-c9e3-468c-a5a4-1677d621559b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:23.462575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9609f72-91d2-4409-b341-da136752d622 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:23.464961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9609f72-91d2-4409-b341-da136752d622 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:23.469300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9609f72-91d2-4409-b341-da136752d622 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:23.472308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 79/153] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:23 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:23.481094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:23.487752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1232fa90-3ce2-4c1f-be02-24b259b672ea None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:23.488628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1232fa90-3ce2-4c1f-be02-24b259b672ea None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:23.489569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1232fa90-3ce2-4c1f-be02-24b259b672ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:23.525243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1232fa90-3ce2-4c1f-be02-24b259b672ea None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:23.528830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1232fa90-3ce2-4c1f-be02-24b259b672ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:23.557100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:23.558298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:23.558504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:23.558759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:23.599942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:23.606276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:23.607768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:23.609097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 75/154] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:23.621352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:23.739181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:23.764981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 80/155] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:23.766354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1232fa90-3ce2-4c1f-be02-24b259b672ea None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:23.914623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd99d36-f216-493f-a4fc-179a8b0813d0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:23.965108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db1744-4974-41a8-a26a-c5469addc1c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:23.966032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db1744-4974-41a8-a26a-c5469addc1c4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:23.966463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db1744-4974-41a8-a26a-c5469addc1c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:23.967032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db1744-4974-41a8-a26a-c5469addc1c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:23.981256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db1744-4974-41a8-a26a-c5469addc1c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b8db1744-4974-41a8-a26a-c5469addc1c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:23.987894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:23.988959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:23.990279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 76/156] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:07:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:23.991654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422610b7-c8c6-4599-8fa2-70b573495a01 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:23.991654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422610b7-c8c6-4599-8fa2-70b573495a01 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:23.991654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422610b7-c8c6-4599-8fa2-70b573495a01 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:23.993619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 77/157] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:23.999382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422610b7-c8c6-4599-8fa2-70b573495a01 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:24.054590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:24.084180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1232fa90-3ce2-4c1f-be02-24b259b672ea None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:24.172907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422610b7-c8c6-4599-8fa2-70b573495a01 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:24.191329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cf50ae-3271-4234-8986-658dcc5d9192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:24.192673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cf50ae-3271-4234-8986-658dcc5d9192 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:24.193506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cf50ae-3271-4234-8986-658dcc5d9192 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:24.194393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cf50ae-3271-4234-8986-658dcc5d9192 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:24.195927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cf50ae-3271-4234-8986-658dcc5d9192 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:24.196917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cf50ae-3271-4234-8986-658dcc5d9192 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:24.200222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cf50ae-3271-4234-8986-658dcc5d9192 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:24.226071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 81/158] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 461 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:24.230701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc68eac1-eccd-441c-98e5-f51ab3f2ac84 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:24.231914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc68eac1-eccd-441c-98e5-f51ab3f2ac84 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:24.232656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc68eac1-eccd-441c-98e5-f51ab3f2ac84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:24.267405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:24.268521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:24.268968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:24.269502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:24.270812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:24.271369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:24.271617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-dc68eac1-eccd-441c-98e5-f51ab3f2ac84 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:24.273108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dc68eac1-eccd-441c-98e5-f51ab3f2ac84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:24.281191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:24.299187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 78/159] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:24.300326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:24.397777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:24.451950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 82/160] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:24.453400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc68eac1-eccd-441c-98e5-f51ab3f2ac84 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:24.502922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:24.518725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29920de-da4f-4cb7-b2f3-86404c0b5f44 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:24.591110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:24.592057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:24.592469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:24.592895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:24.596015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:24.598372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:24.606418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 79/161] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:24 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:24.607833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:24.727627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc68eac1-eccd-441c-98e5-f51ab3f2ac84 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:24.810085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:24.810763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:24.811037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:24.811309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] PATH_INFO: `/v3/users/6648709a4b6d470c80e9b42cb0d345ad` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:24.812764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] RBAC: Authorizing `identity:get_user(user_id=6648709a4b6d470c80e9b42cb0d345ad)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:24.814306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:24.816543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 83/162] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:24 2022] GET /identity/v3/users/6648709a4b6d470c80e9b42cb0d345ad => generated 247 bytes in 363 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:24.817233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:24.871837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe10bad2-7c0f-40ea-8c17-a717303d8bb5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:24.954226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:24.954901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:24.955178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:24.955445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:24.973399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:24.981693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:24.993575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:24.994691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:24.995918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:24.997337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 80/163] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:07:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 389 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:24.997702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:25.069675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:25.144289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d205bc8d-fcd6-42b9-b24c-216ce6c7e22d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:25.261953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:25.262883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:25.263300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:25.263717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:25.330685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af7f7e-ca55-439e-b96a-d32a24685199 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:25.456554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:25.457388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:25.457550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:25.457880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:25.502438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:25.504614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:25.680040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:25.682392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:25.688865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:25.760919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:25.817425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 84/164] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 1001 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:25.818767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:25.996421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 81/165] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 999 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:25.996966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:26.089831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b6f1bf-d644-41a9-8aa7-77f2a60e44c2 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:26.153935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a436f064-7dba-42ef-ae01-7adbf1628453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:26.155043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a436f064-7dba-42ef-ae01-7adbf1628453 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.155513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a436f064-7dba-42ef-ae01-7adbf1628453 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.155988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a436f064-7dba-42ef-ae01-7adbf1628453 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.157257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a436f064-7dba-42ef-ae01-7adbf1628453 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:26.157804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a436f064-7dba-42ef-ae01-7adbf1628453 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:26.160467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a436f064-7dba-42ef-ae01-7adbf1628453 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:26.176021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 85/166] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:07:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:26.177957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab49892-4a71-4e4d-984e-3e927ba2f854 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.178269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab49892-4a71-4e4d-984e-3e927ba2f854 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.178504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab49892-4a71-4e4d-984e-3e927ba2f854 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.180854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 86/167] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:26.181687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab49892-4a71-4e4d-984e-3e927ba2f854 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:26.208866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ce42d9-3329-4039-b611-9a22b831c207 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:26.215743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:26.241828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:26.258229 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:26.258778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.259051 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.259315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.273233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:26.282063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:26.283704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 82/168] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:26.286620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:26.336039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab49892-4a71-4e4d-984e-3e927ba2f854 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:26.382385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61363a3a-7a20-4f9e-8202-871526d37966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:26.383163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61363a3a-7a20-4f9e-8202-871526d37966 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.383449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61363a3a-7a20-4f9e-8202-871526d37966 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.383848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61363a3a-7a20-4f9e-8202-871526d37966 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.385724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61363a3a-7a20-4f9e-8202-871526d37966 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:26.388042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61363a3a-7a20-4f9e-8202-871526d37966 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:26.390370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 87/169] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:26 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:26.392389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41f5b5c-d035-44f0-9448-863bd2cde8ab None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.392816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41f5b5c-d035-44f0-9448-863bd2cde8ab None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.393225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41f5b5c-d035-44f0-9448-863bd2cde8ab None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.397289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 88/170] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:26.397289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46592b18-1e2f-4d03-b034-c95501b036c8 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.397289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46592b18-1e2f-4d03-b034-c95501b036c8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.397289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46592b18-1e2f-4d03-b034-c95501b036c8 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.397932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 89/171] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:26.398925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d359d-bee2-476b-a0a4-7a887b953af2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.399205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d359d-bee2-476b-a0a4-7a887b953af2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.399543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d359d-bee2-476b-a0a4-7a887b953af2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.426296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b08d359d-bee2-476b-a0a4-7a887b953af2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:26.428530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b08d359d-bee2-476b-a0a4-7a887b953af2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:26.442757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb566de-b353-4de6-9792-cede31382711 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:26.500292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:26.501045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.501251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.501514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.502709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:26.504198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:26.505943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 83/172] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:26 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:26.512113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:26.537319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 90/173] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:26.538776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8ad73-1572-4244-870f-a050675fdb14 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.538987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8ad73-1572-4244-870f-a050675fdb14 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.539231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8ad73-1572-4244-870f-a050675fdb14 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.541229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 91/174] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:26.542173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc8ad73-1572-4244-870f-a050675fdb14 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:26.589503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:26.590256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:26.689844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec811af2-29f6-4754-8352-e2a42881f593 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:26.741743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:26.742706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.743124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.743598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.745312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:26.747187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:26.750078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 84/175] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:26 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:26.751572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:26.763867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc8ad73-1572-4244-870f-a050675fdb14 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:26.847195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44834dd8-185d-432e-bb38-68150fff9c00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:26.848306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44834dd8-185d-432e-bb38-68150fff9c00 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.848656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44834dd8-185d-432e-bb38-68150fff9c00 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.849107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44834dd8-185d-432e-bb38-68150fff9c00 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.850217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44834dd8-185d-432e-bb38-68150fff9c00 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:26.850713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44834dd8-185d-432e-bb38-68150fff9c00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:26.852903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44834dd8-185d-432e-bb38-68150fff9c00 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:26.871634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 92/176] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:26.873963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6846794d-69e8-4c36-9d44-9a0893e8a814 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:26.874420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6846794d-69e8-4c36-9d44-9a0893e8a814 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:26.874857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6846794d-69e8-4c36-9d44-9a0893e8a814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:26.904329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6846794d-69e8-4c36-9d44-9a0893e8a814 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:26.906384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6846794d-69e8-4c36-9d44-9a0893e8a814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:26.976156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b729c73-db75-4c6f-acb9-a57c19f8a8d7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:27.009198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:27.039638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:27.040560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:27.041039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:27.041534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:27.042862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:27.043395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:27.047653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 93/177] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:27.047873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:27.048028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65e184-fc01-44d4-9d26-e81b07971700 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:27.048028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65e184-fc01-44d4-9d26-e81b07971700 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:27.048028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65e184-fc01-44d4-9d26-e81b07971700 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:27.057879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 85/178] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:26 2022] GET /identity/v3/role_assignments?user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 608 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:27.064427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:27.078125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:27.091001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-af65e184-fc01-44d4-9d26-e81b07971700 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:27.101833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-af65e184-fc01-44d4-9d26-e81b07971700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:27.302710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41f8be-f966-4508-b2f0-35b773f4fab9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:27.311431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 94/179] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:27.312683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b193e0-0559-45e9-95fb-1c5230bd9632 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:27.313191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b193e0-0559-45e9-95fb-1c5230bd9632 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:27.313668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b193e0-0559-45e9-95fb-1c5230bd9632 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:27.387493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-10b193e0-0559-45e9-95fb-1c5230bd9632 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:27.388806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:27.389634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:27.390817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:27.390817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:27.391644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=6648709a4b6d470c80e9b42cb0d345ad) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:27.392432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:27.393142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-10b193e0-0559-45e9-95fb-1c5230bd9632 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:27.397157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:27.405429 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 86/180] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:27 2022] GET /identity/v3/role_assignments?user.id=6648709a4b6d470c80e9b42cb0d345ad&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:27.406889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:27.547655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:27.566623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 95/181] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:27.568015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b193e0-0559-45e9-95fb-1c5230bd9632 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:27.673041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aabd0a9-43b5-4fe0-8782-4c75b4a3f7a4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:27.690424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:27.750801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:27.752178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:27.753337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:27.753337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:27.785010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b193e0-0559-45e9-95fb-1c5230bd9632 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:27.843740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:27.844839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:27.845422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:27.845887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] PATH_INFO: `/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:27.848558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] RBAC: Authorizing `identity:get_user(user_id=1c1b7dba7cb84f5c89be5d6f9d6f60a7)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:27.850875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:27.854530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 96/182] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:27 2022] GET /identity/v3/users/1c1b7dba7cb84f5c89be5d6f9d6f60a7 => generated 275 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:27.856127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:27.908746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:27.911780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:28.011098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:28.047234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0ffb46-f43b-4ccd-b363-f4263fc8ba1b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:28.100077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 87/183] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 694 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:28.101413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:28.111481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:28.112550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:28.113029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:28.113512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:28.116772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:28.152200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:28.159370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:28.161156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 97/184] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:07:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:28.162759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:28.279720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82dec44-19b3-4208-b243-bb260132e790 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:28.308154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45df65-d1a8-49da-94be-168e124aa43f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:28.329177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:28.329819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:28.330089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:28.330369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:28.346536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:28.355710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:28.357578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:28.359203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:28.360482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:28.360482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:28.360482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:28.360482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:28.360966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 88/185] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:28.360966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:28.372392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:28.380793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:28.389543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:28.391279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 98/186] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:07:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:28.392133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:28.486268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:28.536934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec1bc48-e0c9-4b21-88a0-b1b48bfeabf3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:28.555262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a113886-024a-44e4-922c-cc08c2cad36e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:28.587212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:28.587212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:28.587212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:28.587212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:28.603612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:28.604317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:28.604718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:28.605097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:28.606858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:28.608624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:28.617183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 99/187] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:28.617183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:28.710582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:28.713741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:28.734917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:28.768367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443b6190-0cee-4baa-8a6a-f504289b8af4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:28.803329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:28.823263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:28.823917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:28.824192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:28.824459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:28.825751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:28.827233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:28.828982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 100/188] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:28 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:28.829905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:28.854710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 89/189] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 496 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:28.856642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:28.992671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2936bc4-2056-4eb7-bb5e-4537b6154104 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:28.996879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dede8470-0614-4ca5-931b-717e88bdd5ed None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:29.005523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:29.039138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.040022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.040495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.040958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.041838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:29.042151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.043571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.047561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.048340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.048688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.048958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.049937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:29.050471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.052300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.056518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 101/190] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:29.057931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.067681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 90/191] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:29.068524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.133149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:29.213505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ee9b74-86d8-4997-befc-51cda6fc7a36 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:29.232123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0fb6a7-d848-4621-95ff-118e420d3c2a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:29.271801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.272406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.272661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.272987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.274309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:29.275325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.276782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.283022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.283637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.283915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.284243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.286891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.286891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.287279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 102/192] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:07:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:29.288710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.314961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 91/193] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:07:29 2022] POST /identity/v3/users => generated 297 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:29.316131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.371439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:29.497360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a82474-83e9-45b2-9541-83003a9bbe50 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:29.509103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397263c5-a88f-4a86-a6c1-4413879f9da6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:29.520736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:29.569088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.569742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.570017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.570282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] PATH_INFO: `/v3/projects/8f8d1152ed1e402fade470adfdc1f70f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.572135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] RBAC: Authorizing `identity:get_project(project_id=8f8d1152ed1e402fade470adfdc1f70f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.573944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.575262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.575994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 92/194] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:29 2022] GET /identity/v3/projects/8f8d1152ed1e402fade470adfdc1f70f => generated 312 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:29.576314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.576793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.577040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.577328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.578782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] RBAC: Adding query filter params (role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:29.579311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.584352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.595450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 103/195] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:29 2022] GET /identity/v3/role_assignments?user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 608 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:29.596493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.798338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:29.815956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0116b0-1dda-4f45-9127-28b67dafae1c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:29.816754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64eab9ec-8b58-47a1-9f44-84fa0720a81b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:29.892866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.893608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.893882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.894157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] PATH_INFO: `/v3/users/efb9f072219f40b896cd2c251834dafe` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.898284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] RBAC: Authorizing `identity:get_user(user_id=efb9f072219f40b896cd2c251834dafe)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.900416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.902310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 104/196] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:29 2022] GET /identity/v3/users/efb9f072219f40b896cd2c251834dafe => generated 289 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:29.903232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.908622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:29.910331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:29.911122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:29.911959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] PATH_INFO: `/v3/users/7c116b929a514781aa4ae90359edad2a` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:29.952794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] RBAC: Authorizing `identity:get_user(user_id=7c116b929a514781aa4ae90359edad2a)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:29.956417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:29.960412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 93/197] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:29 2022] GET /identity/v3/users/7c116b929a514781aa4ae90359edad2a => generated 283 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:29.962613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:29.989747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:30.108278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b0b515-7b10-4f26-928b-5e86d2d9bbee None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:30.186666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:30.187735 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:30.188191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:30.188688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] PATH_INFO: `/v3/users/7c116b929a514781aa4ae90359edad2a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:30.190282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] RBAC: Authorizing `identity:get_user(user_id=7c116b929a514781aa4ae90359edad2a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:30.192369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:30.195964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 105/198] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:29 2022] GET /identity/v3/users/7c116b929a514781aa4ae90359edad2a => generated 283 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:30.197994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:30.199046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4017f62c-bd1a-4ba1-858a-8d6540df6a94 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:30.252376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:30.253261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:30.253629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:30.253960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:30.255066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=8f8d1152ed1e402fade470adfdc1f70f, user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:30.255611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:30.257917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:30.268938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 94/199] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:29 2022] GET /identity/v3/role_assignments?user.id=1c1b7dba7cb84f5c89be5d6f9d6f60a7&scope.project.id=8f8d1152ed1e402fade470adfdc1f70f&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 608 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:30.270122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:30.340343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:30.412651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef961c82-3e90-4b4d-9874-e53f34ecbdd0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:30.434577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:30.464698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:30.465267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:30.465544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:30.465813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:30.590825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a03ab3e-70a1-4769-917b-dce518263afe None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:30.612644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:30.614745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:30.651582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:30.652321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:30.652684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:30.653011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:30.783905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:30.786947 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:30.812759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:30.815327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 106/200] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 619 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:30.817469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:30.851924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:30.977567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 95/201] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 707 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:30.979238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:31.031185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233346d1-3344-4a86-9f30-aa9904fe9b37 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:31.099700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:31.101155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:31.102056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:31.102882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:31.175793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a71cd8-86b0-480e-84d0-bbfa477d14f7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:31.200321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:31.236537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:31.239773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:31.241448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:31.242592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:31.244087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:31.244578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:31.246914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:31.249654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:31.252870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 96/202] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:30 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:31.254392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:31.301694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:31.404195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 107/203] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 583 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:31.407281 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:31.463544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333c1d34-acec-4271-9d14-62ec510cfeeb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:31.535099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:31.536069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:31.536545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:31.537026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:31.539159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:31.541986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:31.545499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 97/204] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:31 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:31.546835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:31.580982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cb2350-362e-48fe-a902-62a48e40be0a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:31.609213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:31.654304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:31.657766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:31.659276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:31.659852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:31.660440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:31.663473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] RBAC: Adding query filter params (role.id=b19c6f50e2f542aa81ec4cf175cba448, scope.project.id=abd8cdfe3c4b488eb590b64706ada806, user.id=efb9f072219f40b896cd2c251834dafe) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:31.664315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:31.668196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:31.681670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 108/205] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:31 2022] GET /identity/v3/role_assignments?user.id=efb9f072219f40b896cd2c251834dafe&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=b19c6f50e2f542aa81ec4cf175cba448 => generated 608 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:31.683234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:31.798590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e22aef-14c6-4751-b334-017579b6709b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:31.858667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef76c118-0fc0-4c37-a551-6d659d50243b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:31.900585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:31.908892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:31.908892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:31.908892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:31.912255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:31.914818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:31.922631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 98/206] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:31 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 377 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:31.924311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfa54d5-17e4-461a-af8d-3bf091b01242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:31.925357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfa54d5-17e4-461a-af8d-3bf091b01242 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:31.926025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfa54d5-17e4-461a-af8d-3bf091b01242 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:31.926508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfa54d5-17e4-461a-af8d-3bf091b01242 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:31.927853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfa54d5-17e4-461a-af8d-3bf091b01242 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=7c116b929a514781aa4ae90359edad2a) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:31.928380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfa54d5-17e4-461a-af8d-3bf091b01242 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:31.931787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:31.933571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfa54d5-17e4-461a-af8d-3bf091b01242 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:31.955321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 109/207] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:31 2022] GET /identity/v3/role_assignments?user.id=7c116b929a514781aa4ae90359edad2a&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:32.142028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77f5167-41db-442e-97b7-daaad203c4d6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:32.169154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:32.201087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee835be-7391-4022-991b-3f24d6ff7c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:32.202147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee835be-7391-4022-991b-3f24d6ff7c3d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:32.202622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee835be-7391-4022-991b-3f24d6ff7c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:32.203096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee835be-7391-4022-991b-3f24d6ff7c3d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:32.204457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee835be-7391-4022-991b-3f24d6ff7c3d None admin] RBAC: Adding query filter params (role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=7c116b929a514781aa4ae90359edad2a) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:32.204981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee835be-7391-4022-991b-3f24d6ff7c3d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:32.208071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee835be-7391-4022-991b-3f24d6ff7c3d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:32.217665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 99/208] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:31 2022] GET /identity/v3/role_assignments?user.id=7c116b929a514781aa4ae90359edad2a&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 259 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:32.549390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d2a9b-963a-422b-a13c-f5ec7486106a None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:32.549911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d2a9b-963a-422b-a13c-f5ec7486106a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:32.550201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d2a9b-963a-422b-a13c-f5ec7486106a None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:32.552690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 110/209] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:32.562266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cad28-5379-4291-9272-8584e16cba33 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:32.562713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cad28-5379-4291-9272-8584e16cba33 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:32.563149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cad28-5379-4291-9272-8584e16cba33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:32.592793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9b8cad28-5379-4291-9272-8584e16cba33 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:32.594579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9b8cad28-5379-4291-9272-8584e16cba33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:32.675146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f8947f-b3ca-4544-a509-2ed4675464ca None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:32.675550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f8947f-b3ca-4544-a509-2ed4675464ca None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:32.675949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f8947f-b3ca-4544-a509-2ed4675464ca None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:32.684652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 111/210] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:32.686897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba578e6-e036-4f75-ac13-839aad4e8344 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:32.687503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba578e6-e036-4f75-ac13-839aad4e8344 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:32.687879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba578e6-e036-4f75-ac13-839aad4e8344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:32.716419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-bba578e6-e036-4f75-ac13-839aad4e8344 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:32.718096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bba578e6-e036-4f75-ac13-839aad4e8344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:32.731269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 100/211] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:32.778143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:32.834314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 112/212] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:33.110057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8cad28-5379-4291-9272-8584e16cba33 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:33.248482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba578e6-e036-4f75-ac13-839aad4e8344 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:33.275461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8cad28-5379-4291-9272-8584e16cba33 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:33.302037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:33.344970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:33.346245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:33.346618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:33.346982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:33.368837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:33.376052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:33.377805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:33.378637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:33.383570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 101/213] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:33.387397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:33.401865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba578e6-e036-4f75-ac13-839aad4e8344 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:33.452241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:33.453895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:33.454655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:33.455310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:33.458159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:33.462223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:33.473490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 113/214] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:33.475421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:33.542857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880a80e2-6d43-4846-a584-e6336b714ffb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:33.561205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:33.605259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:33.606424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:33.606937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:33.607388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:33.608544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] RBAC: Adding query filter params (name=member) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:33.609048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:33.611565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:33.628485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 102/215] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:33.629908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:33.654780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd4aceb-7cf4-4d0e-9908-0217a8367531 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:33.675499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:33.707110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:33.708198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:33.708784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:33.709283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:33.710181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:33.712457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:33.744195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 114/216] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:07:33 2022] POST /identity/v3/users => generated 297 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:33.784922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a8ac9d-6bae-4834-b4bc-10798bfa4811 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:33.869998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:33.871861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:33.872706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:33.873499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] PATH_INFO: `/v3/users/ee82cd9e9c0142a1a8670820f2e55f93` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:33.900370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] RBAC: Authorizing `identity:get_user(user_id=ee82cd9e9c0142a1a8670820f2e55f93)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:33.904134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:33.908307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 103/217] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:33 2022] GET /identity/v3/users/ee82cd9e9c0142a1a8670820f2e55f93 => generated 297 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:33.922091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:34.126310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:34.139578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545c9f44-11d5-4768-8115-1f43ff8c4363 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:34.208714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:34.209773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:34.210223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:34.211064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:34.371554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:34.374316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:34.573356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:34.587437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 115/218] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 670 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:34.596245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:34.795736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:34.837060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9cf5b1-5d2b-4cf1-9930-888a456be409 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:34.935474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1533e1b6-c43a-4e8c-a329-aee94abd9e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:34.936679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1533e1b6-c43a-4e8c-a329-aee94abd9e80 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:34.937175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1533e1b6-c43a-4e8c-a329-aee94abd9e80 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:34.937645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1533e1b6-c43a-4e8c-a329-aee94abd9e80 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:34.939819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1533e1b6-c43a-4e8c-a329-aee94abd9e80 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:34.942583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1533e1b6-c43a-4e8c-a329-aee94abd9e80 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:34.950198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 104/219] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:34 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 355 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:34.959692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:35.038094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b211bb1-0764-4461-bedf-ab4523957b2a None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:35.038600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b211bb1-0764-4461-bedf-ab4523957b2a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:35.039135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b211bb1-0764-4461-bedf-ab4523957b2a None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:35.041438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 105/220] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:07:35.059432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7c3717-1ddc-4dd5-b054-0b5f41063723 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:35.059960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7c3717-1ddc-4dd5-b054-0b5f41063723 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:35.060401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7c3717-1ddc-4dd5-b054-0b5f41063723 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:35.119096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3d7c3717-1ddc-4dd5-b054-0b5f41063723 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:35.121296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3d7c3717-1ddc-4dd5-b054-0b5f41063723 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:35.247915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9253780-8476-45a9-86df-94d6cbfbfa51 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:35.298215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 106/221] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:35.346289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c2943-877d-4fb2-99bb-22b4cca0f628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:35.347300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c2943-877d-4fb2-99bb-22b4cca0f628 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:35.347780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c2943-877d-4fb2-99bb-22b4cca0f628 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:35.348256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c2943-877d-4fb2-99bb-22b4cca0f628 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:35.349640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c2943-877d-4fb2-99bb-22b4cca0f628 None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=ee82cd9e9c0142a1a8670820f2e55f93) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:35.350187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c2943-877d-4fb2-99bb-22b4cca0f628 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:35.353944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c2943-877d-4fb2-99bb-22b4cca0f628 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:35.369261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 116/222] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:34 2022] GET /identity/v3/role_assignments?user.id=ee82cd9e9c0142a1a8670820f2e55f93&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 259 bytes in 411 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:35.398450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42ea2bc-071e-4991-ab6f-0e910b328573 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:35.399438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42ea2bc-071e-4991-ab6f-0e910b328573 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:35.400373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42ea2bc-071e-4991-ab6f-0e910b328573 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:35.403498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 107/223] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:35 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:35.411889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d44aab-16b0-489e-8f3f-da26c3aef391 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:35.412306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d44aab-16b0-489e-8f3f-da26c3aef391 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:35.412705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d44aab-16b0-489e-8f3f-da26c3aef391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:35.458847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b8d44aab-16b0-489e-8f3f-da26c3aef391 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:35.460851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b8d44aab-16b0-489e-8f3f-da26c3aef391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:35.593297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:35.624012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 117/224] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:35.703050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42ea2bc-071e-4991-ab6f-0e910b328573 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:35.882633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:35.911706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42ea2bc-071e-4991-ab6f-0e910b328573 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:35.999397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:36.000434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:36.000904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:36.001402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:36.030716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:36.040085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:36.041538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:36.042977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:36.050750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 108/225] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:07:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:36.062904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d44aab-16b0-489e-8f3f-da26c3aef391 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:36.263989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:36.307287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d44aab-16b0-489e-8f3f-da26c3aef391 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:36.373301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:36.374307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:36.374787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:36.375250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:36.376470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:36.376982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:36.379579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:36.412897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 118/226] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:36.424044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:36.572155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:36.587406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b82626-b496-4a91-b7a0-e4cfab681f12 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:36.667225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf5e09-dc3a-4ade-8ccc-4cfaea7e813d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:36.699568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:36.700573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:36.701006 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:36.701494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:36.741541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:36.746296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:36.747668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:36.753770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:36.754486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] PATH_INFO: `/v3/users/7c116b929a514781aa4ae90359edad2a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:36.757111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] RBAC: Authorizing `identity:get_user(user_id=7c116b929a514781aa4ae90359edad2a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:36.759657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:36.761915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:36.763722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:36.765514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:36.769499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 119/227] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:36 2022] GET /identity/v3/users/7c116b929a514781aa4ae90359edad2a => generated 283 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:36.771744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 109/228] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:07:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 509 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:07:36.778288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:36.800317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:36.841793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:37.083598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d14590-53e5-4346-93dd-e90156d2e5d5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:37.144167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08fd542-a646-4087-832b-8babb5085a11 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:37.218138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:37.219148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.219624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.220097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.221336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:37.221872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:37.224504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:37.228323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:37.260387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 110/229] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:07:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 461 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:37.270577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:37.290391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:37.291453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.291936 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.292412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.298042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:37.300783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:37.307266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 120/230] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:36 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 530 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:37.315196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:37.496763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:37.547090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2677b7-c89a-4fb4-8421-5f4d0a0764e3 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:37.573340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be222f52-c732-4ccc-b10a-a8cc0ec9fe39 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:37.607268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:37.608308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.608774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.609261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/7c116b929a514781aa4ae90359edad2a/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.612408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] RBAC: Authorizing `identity:create_grant(project_id=64d8a5429b86463da1df922aeb084e5f, user_id=7c116b929a514781aa4ae90359edad2a, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:37.614807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:37.641041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116c8efd-a468-4db0-a170-090d7d038690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:37.642706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116c8efd-a468-4db0-a170-090d7d038690 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.643372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116c8efd-a468-4db0-a170-090d7d038690 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.644340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116c8efd-a468-4db0-a170-090d7d038690 None admin] PATH_INFO: `/v3/users/7c116b929a514781aa4ae90359edad2a` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.646753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116c8efd-a468-4db0-a170-090d7d038690 None admin] RBAC: Authorizing `identity:get_user(user_id=7c116b929a514781aa4ae90359edad2a)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:37.649773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 121/231] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:37 2022] PUT /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/7c116b929a514781aa4ae90359edad2a/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:37.649961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116c8efd-a468-4db0-a170-090d7d038690 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:37.658950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 111/232] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:37 2022] GET /identity/v3/users/7c116b929a514781aa4ae90359edad2a => generated 283 bytes in 389 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:37.668179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:37.715735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff8f84-7ac3-4cf1-bb2e-c0770ce6009f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.716152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff8f84-7ac3-4cf1-bb2e-c0770ce6009f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.716421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff8f84-7ac3-4cf1-bb2e-c0770ce6009f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.718054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 112/233] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:07:37.729025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9027d533-641d-44ce-aea2-d5fbcd057b70 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.731167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9027d533-641d-44ce-aea2-d5fbcd057b70 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.732392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9027d533-641d-44ce-aea2-d5fbcd057b70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.771221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9027d533-641d-44ce-aea2-d5fbcd057b70 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:37.773276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9027d533-641d-44ce-aea2-d5fbcd057b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:37.777138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:37.888339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad318559-ea70-4595-b8eb-9c0db1c52c6a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:37.957174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6311a4-7e1a-423d-bcaf-4fb4a58cc030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:37.957996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6311a4-7e1a-423d-bcaf-4fb4a58cc030 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.958237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6311a4-7e1a-423d-bcaf-4fb4a58cc030 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.958507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6311a4-7e1a-423d-bcaf-4fb4a58cc030 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.960812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6311a4-7e1a-423d-bcaf-4fb4a58cc030 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:37.962927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6311a4-7e1a-423d-bcaf-4fb4a58cc030 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:37.965618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 122/234] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:37 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:37.967758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9918d0c2-984a-4333-b8dd-7ccc49dcd819 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.968138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9918d0c2-984a-4333-b8dd-7ccc49dcd819 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.968562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9918d0c2-984a-4333-b8dd-7ccc49dcd819 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:37.970475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 123/235] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:37.974837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9918d0c2-984a-4333-b8dd-7ccc49dcd819 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:37.976557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 113/236] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:37.978148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18fb150-8a56-4a58-b775-add03dd02e54 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:37.978148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18fb150-8a56-4a58-b775-add03dd02e54 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:37.978438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18fb150-8a56-4a58-b775-add03dd02e54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:38.005856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f18fb150-8a56-4a58-b775-add03dd02e54 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:38.008154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f18fb150-8a56-4a58-b775-add03dd02e54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:38.151654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:38.163899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 114/237] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:38.205805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9918d0c2-984a-4333-b8dd-7ccc49dcd819 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:38.240437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:38.264175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:38.265414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:38.265867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:38.266436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/7c116b929a514781aa4ae90359edad2a/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:38.270277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] RBAC: Authorizing `identity:create_grant(project_id=64d8a5429b86463da1df922aeb084e5f, user_id=7c116b929a514781aa4ae90359edad2a, role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:38.273219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:38.295737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 124/238] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:37 2022] PUT /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/7c116b929a514781aa4ae90359edad2a/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:38.586215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18fb150-8a56-4a58-b775-add03dd02e54 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:38.711570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:38.841801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18fb150-8a56-4a58-b775-add03dd02e54 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:38.906319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:38.907362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:38.907821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:38.909407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:38.910729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:38.913509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:38.921171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 115/239] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:38.930193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:38.958166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94217e7-9b1b-4fd8-a955-1988a2b5c396 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:39.021756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491e12de-f1b6-4a0b-b070-b8dbcac62300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:39.022631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491e12de-f1b6-4a0b-b070-b8dbcac62300 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:39.023095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491e12de-f1b6-4a0b-b070-b8dbcac62300 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:39.023459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491e12de-f1b6-4a0b-b070-b8dbcac62300 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:39.042261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491e12de-f1b6-4a0b-b070-b8dbcac62300 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-491e12de-f1b6-4a0b-b070-b8dbcac62300 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:39.049057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:39.050073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:39.050909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:39.054950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 125/240] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:39.061623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51982d2c-9c76-4501-9fd4-0bfcdc4e35d9 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:39.061623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51982d2c-9c76-4501-9fd4-0bfcdc4e35d9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:39.061623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51982d2c-9c76-4501-9fd4-0bfcdc4e35d9 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:39.064773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 126/241] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:39.066050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51982d2c-9c76-4501-9fd4-0bfcdc4e35d9 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:39.090121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:39.141698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:39.165587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d8f16-0fc9-448e-a4e2-f30e4ad030f2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:39.219872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326db0b-fa4e-4353-9412-1fdc19f48a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:39.221536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326db0b-fa4e-4353-9412-1fdc19f48a97 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:39.222555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326db0b-fa4e-4353-9412-1fdc19f48a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:39.223362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326db0b-fa4e-4353-9412-1fdc19f48a97 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:39.224775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326db0b-fa4e-4353-9412-1fdc19f48a97 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:39.228342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326db0b-fa4e-4353-9412-1fdc19f48a97 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:39.251753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51982d2c-9c76-4501-9fd4-0bfcdc4e35d9 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:39.268536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 116/242] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:07:38 2022] POST /identity/v3/users => generated 293 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:07:39.275544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725e4857-4785-47a8-96f8-fad2e0d9bf4e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:39.276080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725e4857-4785-47a8-96f8-fad2e0d9bf4e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:39.277605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725e4857-4785-47a8-96f8-fad2e0d9bf4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:39.342975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-725e4857-4785-47a8-96f8-fad2e0d9bf4e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:39.349449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-725e4857-4785-47a8-96f8-fad2e0d9bf4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:39.350537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:39.352757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:39.353258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:39.353709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:39.357983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:39.358507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:39.361851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:39.394581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 127/243] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:39 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:39.403186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:39.584630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 117/244] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:39.657028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:39.663570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03b1d6e-072d-401d-83fb-807e411b6aa1 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:39.721187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:39.721187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:39.721187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:39.721187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] PATH_INFO: `/v3/users/a34dfde0230d41ecbcf86ca69f3457ff` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:39.739337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] RBAC: Authorizing `identity:get_user(user_id=a34dfde0230d41ecbcf86ca69f3457ff)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:39.740977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:39.743469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 128/245] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:39 2022] GET /identity/v3/users/a34dfde0230d41ecbcf86ca69f3457ff => generated 297 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:39.749517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725e4857-4785-47a8-96f8-fad2e0d9bf4e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:39.830038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:39.897682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725e4857-4785-47a8-96f8-fad2e0d9bf4e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:39.951590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:39.952263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:39.952595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:39.952871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:40.054296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:40.056484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:40.087081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:40.186181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:40.285089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 118/246] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 536 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 27 02:07:40.294913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:40.344892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ec0bd-e024-4cfb-9286-0ce631ed5520 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:40.409149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:40.409852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:40.410309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:40.410724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:40.428599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:40.437518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:40.440257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:40.442361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:40.448770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 129/247] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 361 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:40.455887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:40.477995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d0772-2b83-4c5d-addf-cf6bbce406d7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:40.525722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:40.533294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:40.534390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:40.534811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:40.535085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:40.537410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:40.539418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:40.544530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 119/248] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:40 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:40.556013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:40.597466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:40.639537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79e2aa9-3373-494d-9c2a-1e38e68f92e5 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:40.715941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:40.725314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:40.725314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:40.725314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:40.725314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:40.726704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:40.729918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:40.765529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 130/249] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:40.775764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:40.819999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8831ba01-c59a-490e-9f58-fbdaa9785382 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:40.934014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:40.936245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:40.936863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:40.937599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:40.939416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] RBAC: Adding query filter params (role.id=b19c6f50e2f542aa81ec4cf175cba448, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=a34dfde0230d41ecbcf86ca69f3457ff) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:40.940298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:40.943627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:40.965657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 120/250] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:40 2022] GET /identity/v3/role_assignments?user.id=a34dfde0230d41ecbcf86ca69f3457ff&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=b19c6f50e2f542aa81ec4cf175cba448 => generated 259 bytes in 410 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:40.979022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:41.035822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf97b37a-1fec-4680-8f38-d689695244f9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:41.118774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:41.120555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:41.121415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:41.122163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] PATH_INFO: `/v3/users/ee82cd9e9c0142a1a8670820f2e55f93` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:41.124640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] RBAC: Authorizing `identity:get_user(user_id=ee82cd9e9c0142a1a8670820f2e55f93)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:41.127808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:41.133275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 131/251] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:40 2022] GET /identity/v3/users/ee82cd9e9c0142a1a8670820f2e55f93 => generated 297 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:41.142731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:41.307503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:41.340187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cc9ae4-6f82-4cbe-9e9c-4e49a7d5007c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:41.398622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa2e90c-faaa-4295-87be-b64084f8b007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:41.399474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e90c-faaa-4295-87be-b64084f8b007 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:41.399885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e90c-faaa-4295-87be-b64084f8b007 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:41.400273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa2e90c-faaa-4295-87be-b64084f8b007 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:41.402102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa2e90c-faaa-4295-87be-b64084f8b007 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:41.404532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa2e90c-faaa-4295-87be-b64084f8b007 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:41.406987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 121/252] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:41 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:41.415178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:41.549307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae034660-ee51-4070-95e8-5fc0564a6cad None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:41.550002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae034660-ee51-4070-95e8-5fc0564a6cad None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:41.550622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae034660-ee51-4070-95e8-5fc0564a6cad None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:41.552751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 122/253] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:07:41.565615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8bca4-dd56-4549-9c23-5a3755ba8025 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:41.566449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8bca4-dd56-4549-9c23-5a3755ba8025 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:41.567157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8bca4-dd56-4549-9c23-5a3755ba8025 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:41.596961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d7d12d-9088-48c0-8992-81a242468971 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:41.605961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-21d8bca4-dd56-4549-9c23-5a3755ba8025 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:41.608000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-21d8bca4-dd56-4549-9c23-5a3755ba8025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:41.688253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f1cf4-59c8-4700-9822-7012dd51e43e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:41.689858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f1cf4-59c8-4700-9822-7012dd51e43e None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:41.690786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f1cf4-59c8-4700-9822-7012dd51e43e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:41.692730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f1cf4-59c8-4700-9822-7012dd51e43e None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/ee82cd9e9c0142a1a8670820f2e55f93/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:41.698104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f1cf4-59c8-4700-9822-7012dd51e43e None admin] RBAC: Authorizing `identity:create_grant(project_id=64d8a5429b86463da1df922aeb084e5f, user_id=ee82cd9e9c0142a1a8670820f2e55f93, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:41.701535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f1cf4-59c8-4700-9822-7012dd51e43e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:41.732003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 132/254] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:41 2022] PUT /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/ee82cd9e9c0142a1a8670820f2e55f93/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:07:41.740690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed78c28-a36b-4df0-a551-bf7ecce63633 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:41.741623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed78c28-a36b-4df0-a551-bf7ecce63633 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:41.742363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed78c28-a36b-4df0-a551-bf7ecce63633 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:41.745817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 133/255] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:41 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:41.755009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4503d6e1-e233-4f41-81f8-42d06f2cede6 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:41.756033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4503d6e1-e233-4f41-81f8-42d06f2cede6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:41.756830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4503d6e1-e233-4f41-81f8-42d06f2cede6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:41.791369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:41.814949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4503d6e1-e233-4f41-81f8-42d06f2cede6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:41.817721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4503d6e1-e233-4f41-81f8-42d06f2cede6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:41.871805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 123/256] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:41.987969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 134/257] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:42.563636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d8bca4-dd56-4549-9c23-5a3755ba8025 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:42.680526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4503d6e1-e233-4f41-81f8-42d06f2cede6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:42.795773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:42.824597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d8bca4-dd56-4549-9c23-5a3755ba8025 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:42.873974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4503d6e1-e233-4f41-81f8-42d06f2cede6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:42.914364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:42.915298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:42.915722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:42.916272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:42.935001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:42.936527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:42.937373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:42.938033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:42.944281 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:42.956195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:42.963606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:42.965365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:42.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 124/258] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:07:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 405 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:42.986154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:42.992132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:42.996849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:42.998954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:43.000245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:43.002118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 135/259] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:07:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:07:43.007135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:43.182460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a894f7-aa3b-4f02-b876-1aa92ee1b450 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:43.217698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f950bf-5d78-4b41-a133-6e1727fa3e8a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:43.241274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:43.242346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:43.242765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:43.243198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:43.244453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:43.244918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:43.247115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:43.262067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 136/260] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:07:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:43.270349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:43.286847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:43.287751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:43.288289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:43.288566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:43.289321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:43.289634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:43.291601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:43.316692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 125/261] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:43.328380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:43.361691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:43.494477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:43.556620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ed371-1876-400d-9343-4ca6f5ceb6c4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:43.567027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b06136-5672-4352-9afa-e27643d18663 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:43.603040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:43.609156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:43.609156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:43.609156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] PATH_INFO: `/v3/users/7c116b929a514781aa4ae90359edad2a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:43.609156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] RBAC: Authorizing `identity:get_user(user_id=7c116b929a514781aa4ae90359edad2a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:43.609613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:43.612688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 137/262] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:43 2022] GET /identity/v3/users/7c116b929a514781aa4ae90359edad2a => generated 283 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:43.621078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:43.673465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:43.674638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:43.675159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:43.675858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] PATH_INFO: `/v3/users/7c116b929a514781aa4ae90359edad2a` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:43.679020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] RBAC: Authorizing `identity:get_user(user_id=7c116b929a514781aa4ae90359edad2a)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:43.687297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:43.696167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 126/263] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:43 2022] GET /identity/v3/users/7c116b929a514781aa4ae90359edad2a => generated 283 bytes in 366 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:43.699884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:43.788073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8596bbf-afd7-4f5f-87be-02e75783b5b4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:43.828484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:43.839807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:43.840796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:43.841293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:43.841713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:43.855189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:43.907813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd2d307-19b9-4289-b02e-35e036c644f3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:43.991754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1cb6c3-1fb7-469e-88b7-8aecb251f6f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:43.996381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cb6c3-1fb7-469e-88b7-8aecb251f6f5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:43.997148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cb6c3-1fb7-469e-88b7-8aecb251f6f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:43.997834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cb6c3-1fb7-469e-88b7-8aecb251f6f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:44.013089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:44.016777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:44.241366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1cb6c3-1fb7-469e-88b7-8aecb251f6f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:44.243892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1cb6c3-1fb7-469e-88b7-8aecb251f6f5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:44.349708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 138/264] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 730 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:44.358566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:44.465731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:44.481609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:44.497251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 127/265] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 798 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 27 02:07:44.499655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a1e5a-b993-4c68-857d-4adc600212e4 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:44.500188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a1e5a-b993-4c68-857d-4adc600212e4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:44.500882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a1e5a-b993-4c68-857d-4adc600212e4 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:44.503696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 128/266] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:44 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:44.513307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a1e5a-b993-4c68-857d-4adc600212e4 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:44.716811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a1e5a-b993-4c68-857d-4adc600212e4 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:44.779123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c15edf-5d24-494e-8120-d53329ce8554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:44.780139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c15edf-5d24-494e-8120-d53329ce8554 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:44.780669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c15edf-5d24-494e-8120-d53329ce8554 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:44.781403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c15edf-5d24-494e-8120-d53329ce8554 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:44.783168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c15edf-5d24-494e-8120-d53329ce8554 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:44.785713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c15edf-5d24-494e-8120-d53329ce8554 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:44.788826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 129/267] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:44 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:44.789308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3851d9-fff9-4377-8580-643ff33d4182 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:44.790686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdd92d4-6f85-441b-bdec-909f6f21df35 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:44.791261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdd92d4-6f85-441b-bdec-909f6f21df35 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:44.791749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdd92d4-6f85-441b-bdec-909f6f21df35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:44.819888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8cdd92d4-6f85-441b-bdec-909f6f21df35 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:44.821139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8cdd92d4-6f85-441b-bdec-909f6f21df35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:44.844001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b9e1ae-ed0f-4e9e-977e-efe6bc290912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:44.846023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9e1ae-ed0f-4e9e-977e-efe6bc290912 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:44.846881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9e1ae-ed0f-4e9e-977e-efe6bc290912 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:44.847983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9e1ae-ed0f-4e9e-977e-efe6bc290912 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:44.852888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b9e1ae-ed0f-4e9e-977e-efe6bc290912 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:44.857506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b9e1ae-ed0f-4e9e-977e-efe6bc290912 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:44.861090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 139/268] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:44 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 503 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:44.862385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b183bff0-9c81-4883-8fe9-88838bd1d82d None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:44.862658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b183bff0-9c81-4883-8fe9-88838bd1d82d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:44.862922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b183bff0-9c81-4883-8fe9-88838bd1d82d None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:44.869916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 140/269] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:44.871688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b183bff0-9c81-4883-8fe9-88838bd1d82d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:44.883622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:44.940374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 130/270] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:44.941926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdd92d4-6f85-441b-bdec-909f6f21df35 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:45.042843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b183bff0-9c81-4883-8fe9-88838bd1d82d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:45.045612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:45.087531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdd92d4-6f85-441b-bdec-909f6f21df35 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:45.103185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad96e94-6c0d-4c2c-bf18-ba36732b069a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:45.104692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad96e94-6c0d-4c2c-bf18-ba36732b069a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:45.105550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad96e94-6c0d-4c2c-bf18-ba36732b069a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:45.106289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad96e94-6c0d-4c2c-bf18-ba36732b069a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:45.107554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad96e94-6c0d-4c2c-bf18-ba36732b069a None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=7c116b929a514781aa4ae90359edad2a) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:45.108106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad96e94-6c0d-4c2c-bf18-ba36732b069a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:45.110275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad96e94-6c0d-4c2c-bf18-ba36732b069a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:45.119803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 141/271] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:44 2022] GET /identity/v3/role_assignments?user.id=7c116b929a514781aa4ae90359edad2a&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:45.122449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0daa2c-d43c-4181-a29f-90f130f3dbd6 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:45.123216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0daa2c-d43c-4181-a29f-90f130f3dbd6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:45.124114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0daa2c-d43c-4181-a29f-90f130f3dbd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:45.147356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45438ae1-1595-4f39-bffa-99ed6ab4e7c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:45.148281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45438ae1-1595-4f39-bffa-99ed6ab4e7c1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:45.148740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45438ae1-1595-4f39-bffa-99ed6ab4e7c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:45.149311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45438ae1-1595-4f39-bffa-99ed6ab4e7c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:45.150555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45438ae1-1595-4f39-bffa-99ed6ab4e7c1 None admin] RBAC: Adding query filter params (role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=7c116b929a514781aa4ae90359edad2a) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:45.150988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45438ae1-1595-4f39-bffa-99ed6ab4e7c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:45.153615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45438ae1-1595-4f39-bffa-99ed6ab4e7c1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:45.156046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-da0daa2c-d43c-4181-a29f-90f130f3dbd6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:45.158379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-da0daa2c-d43c-4181-a29f-90f130f3dbd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:45.165697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 131/272] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:44 2022] GET /identity/v3/role_assignments?user.id=7c116b929a514781aa4ae90359edad2a&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 608 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:45.167583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ec5db-0c39-4e19-9cda-58e3e6e2cde5 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:45.174112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ec5db-0c39-4e19-9cda-58e3e6e2cde5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:45.177710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ec5db-0c39-4e19-9cda-58e3e6e2cde5 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:45.180408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 132/273] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:45 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:45.190814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ecba44-59ff-4120-915c-280730d8ebe1 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:45.191561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ecba44-59ff-4120-915c-280730d8ebe1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:45.192073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ecba44-59ff-4120-915c-280730d8ebe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:45.246096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-51ecba44-59ff-4120-915c-280730d8ebe1 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:45.249528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-51ecba44-59ff-4120-915c-280730d8ebe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:45.345158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:45.365922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 142/274] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:45.374244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0daa2c-d43c-4181-a29f-90f130f3dbd6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:45.447235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 133/275] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:45.556408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:45.575499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0daa2c-d43c-4181-a29f-90f130f3dbd6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:45.651173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:45.652190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:45.652605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:45.653102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] PATH_INFO: `/v3/roles/member` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:45.683754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:45.691847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:45.693692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:45.695071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 143/276] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:45.700998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ecba44-59ff-4120-915c-280730d8ebe1 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:45.874454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:45.909527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ecba44-59ff-4120-915c-280730d8ebe1 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:45.976724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:45.978245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:45.978804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:45.979083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:45.979809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] RBAC: Adding query filter params (name=member) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:45.980117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:45.981563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:45.993532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 134/277] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:45.994394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:46.035535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:46.050949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34092f7-4125-4cd9-bfaa-c1b8d490f0ca None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:46.104338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8c6349-e0cf-41c6-8827-c41e5484203a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:46.104999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8c6349-e0cf-41c6-8827-c41e5484203a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.105301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8c6349-e0cf-41c6-8827-c41e5484203a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.105564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8c6349-e0cf-41c6-8827-c41e5484203a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.106794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8c6349-e0cf-41c6-8827-c41e5484203a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:46.108288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8c6349-e0cf-41c6-8827-c41e5484203a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:46.110197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 144/278] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:46.112448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06514b99-b60e-46c4-977a-bc817144251b None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.112756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06514b99-b60e-46c4-977a-bc817144251b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.113268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06514b99-b60e-46c4-977a-bc817144251b None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.115894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 145/279] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:46.117155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06514b99-b60e-46c4-977a-bc817144251b None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:46.141663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:46.146450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df1f7a-2c9f-4d4a-98a8-4ceb15c011da None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:46.200788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:46.201576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.202018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.202348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.219430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:46.228048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:46.229211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:46.230066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 135/280] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:46.231074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:46.262211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06514b99-b60e-46c4-977a-bc817144251b None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:46.323546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5abea2-55b8-4d0d-94f9-d7c610198f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:46.323920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5abea2-55b8-4d0d-94f9-d7c610198f77 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.324284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5abea2-55b8-4d0d-94f9-d7c610198f77 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.324715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5abea2-55b8-4d0d-94f9-d7c610198f77 None admin] PATH_INFO: `/v3/users/109ba80a80b74e48a8c17505d4be49ce` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.349365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5abea2-55b8-4d0d-94f9-d7c610198f77 None admin] RBAC: Authorizing `identity:get_user(user_id=109ba80a80b74e48a8c17505d4be49ce)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:46.351853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5abea2-55b8-4d0d-94f9-d7c610198f77 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:46.354724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 146/281] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:46 2022] GET /identity/v3/users/109ba80a80b74e48a8c17505d4be49ce => generated 293 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:46.356491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a0b70c-6288-4e39-88a0-bf35eada9506 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.356895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a0b70c-6288-4e39-88a0-bf35eada9506 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.357243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a0b70c-6288-4e39-88a0-bf35eada9506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.381457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:46.381898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b8a0b70c-6288-4e39-88a0-bf35eada9506 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:46.383725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b8a0b70c-6288-4e39-88a0-bf35eada9506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:46.399760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92518aba-ddb4-4177-81ef-7dcf4c4a1c92 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:46.456902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:46.457872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.458388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.458860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.460081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:46.462874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:46.492944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 147/282] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:46.494490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a0b70c-6288-4e39-88a0-bf35eada9506 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:46.497277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 136/283] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:07:46 2022] POST /identity/v3/users => generated 293 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:46.499238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:46.507991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:46.699946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66828293-62e8-4087-a750-549af2e728d3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:46.722849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a0b70c-6288-4e39-88a0-bf35eada9506 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:46.765166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:46.765166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.765166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.765166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.765786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, user.id=109ba80a80b74e48a8c17505d4be49ce, scope.system=all) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:46.765786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:46.768626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:46.775450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 137/284] 10.222.0.52 () {56 vars in 1308 bytes} [Sat Aug 27 02:07:46 2022] GET /identity/v3/role_assignments?user.id=109ba80a80b74e48a8c17505d4be49ce&scope.system=all&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 226 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:46.806562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:46.807297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:46.807577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:46.807849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:46.808563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:46.808861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:46.812460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:46.836999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 148/285] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:46 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:46.845301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:46.862553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:46.894811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:47.023835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:47.026432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f34ea-eaa4-441b-bed4-904dce2cb02b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:47.083809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04e52e8-7f97-4b56-8168-b1403d618c2b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:47.090588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:47.091782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:47.092281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:47.092927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] PATH_INFO: `/v3/users/a34dfde0230d41ecbcf86ca69f3457ff` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:47.095226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] RBAC: Authorizing `identity:get_user(user_id=a34dfde0230d41ecbcf86ca69f3457ff)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:47.098099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:47.104901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 138/286] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:46 2022] GET /identity/v3/users/a34dfde0230d41ecbcf86ca69f3457ff => generated 297 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:47.111271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:47.141547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:47.142195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:47.142469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:47.142736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:47.155364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:47.164590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:47.166716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:47.168268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 149/287] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:07:47.177255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:47.315126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645d64f6-1c5d-4714-98f0-d911cf3fae1c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:47.317727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:47.360995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:47.362099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:47.362581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:47.363039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:47.363913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:47.364617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:47.369452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:47.370672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:47.401235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7146ea-3f97-4bc0-a2a1-a6b5980cb9bb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:47.403012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 150/288] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:47.408087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:47.509850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:47.510532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:47.510806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:47.511220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:47.513261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:47.515756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:47.527143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 139/289] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:47 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 417 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:47.537575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:47.597687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb9d8e-4aa4-4fda-86b3-3183ad8b56bc None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:47.651928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:47.652673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:47.652947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:47.653242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] PATH_INFO: `/v3/users/ee82cd9e9c0142a1a8670820f2e55f93` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:47.654427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] RBAC: Authorizing `identity:get_user(user_id=ee82cd9e9c0142a1a8670820f2e55f93)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:47.656678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:47.659922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 151/290] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:47 2022] GET /identity/v3/users/ee82cd9e9c0142a1a8670820f2e55f93 => generated 297 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:47.668190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:47.748196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f794447-6a6a-4a1e-b252-9fbaceaed760 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:47.777815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:47.857568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:47.858288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:47.858747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:47.859179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/a34dfde0230d41ecbcf86ca69f3457ff/roles/b19c6f50e2f542aa81ec4cf175cba448` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:47.866701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] RBAC: Authorizing `identity:create_grant(project_id=64d8a5429b86463da1df922aeb084e5f, user_id=a34dfde0230d41ecbcf86ca69f3457ff, role_id=b19c6f50e2f542aa81ec4cf175cba448)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:47.869408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:47.876563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1521b7b-b683-4c73-9cf5-ce1043b22d95 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:47.904420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 140/291] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:07:47 2022] PUT /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f/users/a34dfde0230d41ecbcf86ca69f3457ff/roles/b19c6f50e2f542aa81ec4cf175cba448 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:47.949069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:47.950009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:47.950430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:47.950865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:48.094567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:48.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:48.189417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:48.261364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 152/292] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 594 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:48.267251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:48.275642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:48.473091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b6563e-8e37-4d37-b06f-ab916c8e68dc None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:48.525629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f652f-6011-48d4-952f-22f8825225ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:48.526265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f652f-6011-48d4-952f-22f8825225ba None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:48.526536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f652f-6011-48d4-952f-22f8825225ba None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:48.526797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f652f-6011-48d4-952f-22f8825225ba None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:48.528074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f652f-6011-48d4-952f-22f8825225ba None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:48.530792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f652f-6011-48d4-952f-22f8825225ba None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:48.532912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 141/293] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:48 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:48.539388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:48.692184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d045d-4fbf-488c-a0c4-b27d17e5ae92 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:48.768287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d751ddd2-d71a-4ff3-8a73-d1392a287168 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:48.769218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d751ddd2-d71a-4ff3-8a73-d1392a287168 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:48.769656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d751ddd2-d71a-4ff3-8a73-d1392a287168 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:48.770322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d751ddd2-d71a-4ff3-8a73-d1392a287168 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:48.771957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d751ddd2-d71a-4ff3-8a73-d1392a287168 None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=ee82cd9e9c0142a1a8670820f2e55f93) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:48.772446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d751ddd2-d71a-4ff3-8a73-d1392a287168 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:48.774965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d751ddd2-d71a-4ff3-8a73-d1392a287168 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:48.784238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 153/294] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:48 2022] GET /identity/v3/role_assignments?user.id=ee82cd9e9c0142a1a8670820f2e55f93&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 608 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:49.775920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf06a6-0a5d-4dbf-b321-0922cc0aeb6d None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:49.776442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf06a6-0a5d-4dbf-b321-0922cc0aeb6d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:49.776784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bf06a6-0a5d-4dbf-b321-0922cc0aeb6d None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:49.779249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 142/295] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:07:49.787947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1a6df-b64f-4270-abb1-4e50037f4490 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:49.788759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1a6df-b64f-4270-abb1-4e50037f4490 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:49.788759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1a6df-b64f-4270-abb1-4e50037f4490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:49.814876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1bb1a6df-b64f-4270-abb1-4e50037f4490 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:49.816129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1bb1a6df-b64f-4270-abb1-4e50037f4490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:49.843451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:49.920271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 154/296] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:49.960550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cfdaf2-bd35-421b-af62-82a73c8a84fc None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:49.960804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cfdaf2-bd35-421b-af62-82a73c8a84fc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:49.961181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cfdaf2-bd35-421b-af62-82a73c8a84fc None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:49.963331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 143/297] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:49.973095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1571f-8ae0-481d-b042-e22644558f59 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:49.973899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1571f-8ae0-481d-b042-e22644558f59 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:49.974555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd1571f-8ae0-481d-b042-e22644558f59 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:49.975121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33795a4b-55c6-42ed-a67c-cf1feb917552 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:49.975638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33795a4b-55c6-42ed-a67c-cf1feb917552 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:49.976145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33795a4b-55c6-42ed-a67c-cf1feb917552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:49.978309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 155/298] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:49 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:49.987496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cfbb9c-8bfd-409a-afc5-bb39578533bd None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:49.987937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cfbb9c-8bfd-409a-afc5-bb39578533bd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:49.988400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cfbb9c-8bfd-409a-afc5-bb39578533bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:50.000798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-33795a4b-55c6-42ed-a67c-cf1feb917552 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:50.002383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-33795a4b-55c6-42ed-a67c-cf1feb917552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:50.025483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d9cfbb9c-8bfd-409a-afc5-bb39578533bd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:50.028484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d9cfbb9c-8bfd-409a-afc5-bb39578533bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:50.096951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 144/299] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:50.166732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 156/300] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:50.387790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33795a4b-55c6-42ed-a67c-cf1feb917552 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:50.395386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:50.469255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cfbb9c-8bfd-409a-afc5-bb39578533bd None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:50.558698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33795a4b-55c6-42ed-a67c-cf1feb917552 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:50.618213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:50.619199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:50.619628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:50.620056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:50.622414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:50.624879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:50.628686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 145/301] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:50.630352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:50.632879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:50.672212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cfbb9c-8bfd-409a-afc5-bb39578533bd None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:50.734587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:50.735423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:50.735831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:50.736230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:50.741199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:50.751590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:50.757973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:50.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 157/302] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:50.761407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:50.807925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425860f4-b0e2-4e67-badb-73b52d081569 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:50.871415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:50.872473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:50.872946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:50.873447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:50.895090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:50.906396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:50.907968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:50.909408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 146/303] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:50.910585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:50.946707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c224f7f3-b8e1-45fb-8c3c-fff7cfe3650e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:51.010004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:51.011035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.011356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.011805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] PATH_INFO: `/v3/groups` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.012874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:51.015203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:51.033602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:51.047542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 158/304] 10.222.0.52 () {60 vars in 1134 bytes} [Sat Aug 27 02:07:50 2022] POST /identity/v3/groups => generated 229 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:51.053951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:51.100929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f27c1f-48f8-4f11-b338-c4a0ed9daac5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:51.166794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:51.187231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:51.188882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.190350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.190991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.193194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:51.193725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:51.196136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:51.217493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 147/305] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:51.223472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:51.256897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66837ebc-49fa-415e-9ddd-fdbc64099bef None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:51.323031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19b6fd6-b136-4865-980f-a162ab18cf02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:51.323801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19b6fd6-b136-4865-980f-a162ab18cf02 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.324080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19b6fd6-b136-4865-980f-a162ab18cf02 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.324369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19b6fd6-b136-4865-980f-a162ab18cf02 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.325647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19b6fd6-b136-4865-980f-a162ab18cf02 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:51.326191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19b6fd6-b136-4865-980f-a162ab18cf02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:51.329459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19b6fd6-b136-4865-980f-a162ab18cf02 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:51.347840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 159/306] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:51.350988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baebd8e8-2dee-4814-ae03-709e8113b480 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.351483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baebd8e8-2dee-4814-ae03-709e8113b480 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.351890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baebd8e8-2dee-4814-ae03-709e8113b480 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.357152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 160/307] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:51.357152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baebd8e8-2dee-4814-ae03-709e8113b480 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:51.415493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f9ed3d-d231-4491-8152-f4aedae92857 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:51.462848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:51.476795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a81aa1-6234-43c6-b001-401b86e0c154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:51.477384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a81aa1-6234-43c6-b001-401b86e0c154 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.477658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a81aa1-6234-43c6-b001-401b86e0c154 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.477923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a81aa1-6234-43c6-b001-401b86e0c154 None admin] PATH_INFO: `/v3/users/01b5dc097a31474895169eda2fcda733` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.494873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a81aa1-6234-43c6-b001-401b86e0c154 None admin] RBAC: Authorizing `identity:get_user(user_id=01b5dc097a31474895169eda2fcda733)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:51.497480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a81aa1-6234-43c6-b001-401b86e0c154 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:51.500262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 148/308] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:51 2022] GET /identity/v3/users/01b5dc097a31474895169eda2fcda733 => generated 293 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:51.502337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9146baf1-c535-449a-a065-7449c698d820 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.502791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9146baf1-c535-449a-a065-7449c698d820 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.503225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9146baf1-c535-449a-a065-7449c698d820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.526287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9146baf1-c535-449a-a065-7449c698d820 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:51.527901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9146baf1-c535-449a-a065-7449c698d820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:51.538227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baebd8e8-2dee-4814-ae03-709e8113b480 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:51.589213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:51.589986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.590261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.590533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] PATH_INFO: `/v3/users/109ba80a80b74e48a8c17505d4be49ce` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.592076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] RBAC: Authorizing `identity:get_user(user_id=109ba80a80b74e48a8c17505d4be49ce)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:51.593603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:51.595295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 161/309] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:51 2022] GET /identity/v3/users/109ba80a80b74e48a8c17505d4be49ce => generated 293 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:51.596170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:51.603102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:51.636932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 149/310] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:51.639831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9146baf1-c535-449a-a065-7449c698d820 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:51.814858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba39d84c-002c-4ef7-a380-f193f9198152 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:51.869605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9146baf1-c535-449a-a065-7449c698d820 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:51.882903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:51.895195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:51.896533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.897266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.897968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.899873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] RBAC: Adding query filter params (role.id=b19c6f50e2f542aa81ec4cf175cba448, user.id=01b5dc097a31474895169eda2fcda733, scope.system=all) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:51.900610 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:51.903889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:51.916498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 162/311] 10.222.0.52 () {56 vars in 1308 bytes} [Sat Aug 27 02:07:51 2022] GET /identity/v3/role_assignments?user.id=01b5dc097a31474895169eda2fcda733&scope.system=all&role.id=b19c6f50e2f542aa81ec4cf175cba448 => generated 226 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:51.937465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:51.938386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:51.938765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:51.939135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] PATH_INFO: `/v3/system/users/109ba80a80b74e48a8c17505d4be49ce/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:51.941225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=109ba80a80b74e48a8c17505d4be49ce, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:51.944322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:51.963929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 150/312] 10.222.0.52 () {58 vars in 1334 bytes} [Sat Aug 27 02:07:51 2022] PUT /identity/v3/system/users/109ba80a80b74e48a8c17505d4be49ce/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:07:52.010555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:52.171885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3993b262-1616-4bb2-aac7-7ad10f203443 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:52.221899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:52.222918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:52.223334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:52.223876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:52.241164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:52.248817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:52.252141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:52.255948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 163/313] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:07:52.266834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:52.329164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:52.399519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048307df-d0fb-4ec8-a8bb-849cb77eee2c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:52.448258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:52.448783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:52.449111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:52.449395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:52.450148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:52.450589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:52.452073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:52.462967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 151/314] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:52.469917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:52.563742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:52.634080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ba740-1e36-4d35-8d11-f2528c4d981e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:52.698182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:52.699075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:52.699407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:52.699690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] PATH_INFO: `/v3/users/a34dfde0230d41ecbcf86ca69f3457ff` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:52.701460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] RBAC: Authorizing `identity:get_user(user_id=a34dfde0230d41ecbcf86ca69f3457ff)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:52.704021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:52.713294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 164/315] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:52 2022] GET /identity/v3/users/a34dfde0230d41ecbcf86ca69f3457ff => generated 297 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:52.716164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:52.857304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdb211a-196c-4286-aef4-955c691af39f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:52.881565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:52.908079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:52.908961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:52.909268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:52.909542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:53.015043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:53.016598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:53.147259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 152/316] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:07:53.156452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:53.330436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d345d672-2f98-4994-9ecc-3798a98b442a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:53.370850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:53.391099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64505007-a0bd-43b7-9aef-7fa6d2ff8832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:53.392663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64505007-a0bd-43b7-9aef-7fa6d2ff8832 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:53.393383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64505007-a0bd-43b7-9aef-7fa6d2ff8832 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:53.394045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64505007-a0bd-43b7-9aef-7fa6d2ff8832 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:53.395950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64505007-a0bd-43b7-9aef-7fa6d2ff8832 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:53.397981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64505007-a0bd-43b7-9aef-7fa6d2ff8832 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:53.405235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 165/317] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:07:53 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:53.407959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:53.425403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:53.593530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7675ab-033b-400c-af77-1ddf7184e2d4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:53.654275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b4ef48-a731-43b0-af62-be22117b7085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:53.656388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b4ef48-a731-43b0-af62-be22117b7085 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:53.657201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b4ef48-a731-43b0-af62-be22117b7085 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:53.657958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b4ef48-a731-43b0-af62-be22117b7085 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:53.659639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b4ef48-a731-43b0-af62-be22117b7085 None admin] RBAC: Adding query filter params (role.id=b19c6f50e2f542aa81ec4cf175cba448, scope.project.id=64d8a5429b86463da1df922aeb084e5f, user.id=a34dfde0230d41ecbcf86ca69f3457ff) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:53.661418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b4ef48-a731-43b0-af62-be22117b7085 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:53.664463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b4ef48-a731-43b0-af62-be22117b7085 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:53.681236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 153/318] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:07:53 2022] GET /identity/v3/role_assignments?user.id=a34dfde0230d41ecbcf86ca69f3457ff&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=b19c6f50e2f542aa81ec4cf175cba448 => generated 608 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:53.896023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3940b4e-5199-4bad-885d-0319665e5082 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:53.897031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3940b4e-5199-4bad-885d-0319665e5082 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:53.897831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3940b4e-5199-4bad-885d-0319665e5082 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:53.900688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 166/319] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:53 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:53.907114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b99b3eb-a924-4d75-b1d8-759fe45a284c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:53.908060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b99b3eb-a924-4d75-b1d8-759fe45a284c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:53.908361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b99b3eb-a924-4d75-b1d8-759fe45a284c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:53.932108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8b99b3eb-a924-4d75-b1d8-759fe45a284c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:53.933775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8b99b3eb-a924-4d75-b1d8-759fe45a284c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:54.042260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:54.048872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 154/320] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:54.409302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3940b4e-5199-4bad-885d-0319665e5082 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:54.583195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3940b4e-5199-4bad-885d-0319665e5082 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:54.591449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b126fa-e6e9-46bf-8af7-3e822c211a45 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:54.591879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b126fa-e6e9-46bf-8af7-3e822c211a45 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:54.592293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b126fa-e6e9-46bf-8af7-3e822c211a45 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:54.594999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 155/321] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:54.603842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee20242-4a09-4183-8974-e9131bc4cc93 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:54.604228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee20242-4a09-4183-8974-e9131bc4cc93 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:54.604584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee20242-4a09-4183-8974-e9131bc4cc93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:54.634948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9ee20242-4a09-4183-8974-e9131bc4cc93 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:54.636165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9ee20242-4a09-4183-8974-e9131bc4cc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:54.640555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2af0d7-66ed-4cc6-ac02-5b16be7ed53d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:54.641230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2af0d7-66ed-4cc6-ac02-5b16be7ed53d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:54.645905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2af0d7-66ed-4cc6-ac02-5b16be7ed53d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:54.645905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2af0d7-66ed-4cc6-ac02-5b16be7ed53d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:54.645905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2af0d7-66ed-4cc6-ac02-5b16be7ed53d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:54.645905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2af0d7-66ed-4cc6-ac02-5b16be7ed53d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:54.646909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 167/322] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:07:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:54.653784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbbc30-9974-41af-a788-55553c7a9392 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:54.654279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbbc30-9974-41af-a788-55553c7a9392 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:54.654764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbbc30-9974-41af-a788-55553c7a9392 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:54.657589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 168/323] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:54 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:54.658717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacbbc30-9974-41af-a788-55553c7a9392 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:54.746630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 156/324] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:07:54.748316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd665d9-4e88-4f63-9cf7-06c9cade3a01 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:54.748662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd665d9-4e88-4f63-9cf7-06c9cade3a01 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:54.748922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd665d9-4e88-4f63-9cf7-06c9cade3a01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:54.757390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:54.783002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-6fd665d9-4e88-4f63-9cf7-06c9cade3a01 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:54.783453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6fd665d9-4e88-4f63-9cf7-06c9cade3a01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:54.836472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacbbc30-9974-41af-a788-55553c7a9392 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:54.896059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 157/325] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:54.898568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:54.899430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:54.899901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:54.900382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] PATH_INFO: `/v3/groups` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:54.901290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:54.903000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:54.922184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 169/326] 10.222.0.52 () {60 vars in 1134 bytes} [Sat Aug 27 02:07:54 2022] POST /identity/v3/groups => generated 226 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:07:55.148704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd665d9-4e88-4f63-9cf7-06c9cade3a01 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:55.246618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:55.296699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:55.319177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd665d9-4e88-4f63-9cf7-06c9cade3a01 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:55.390001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:55.390963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:55.391422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:55.391820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:55.415164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:55.420108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:55.422169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:55.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 158/327] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:55.433285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:55.482667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31206657-5847-4ac3-a186-38fde1777e02 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:55.517520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:55.558678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:55.559425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:55.559717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:55.559999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:55.578558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:55.587676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:55.588919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:07:55.589753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:07:55.593392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 170/328] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:55.596210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:55.647707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea203cf-fb24-4555-b230-395234d822a4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:55.700043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:55.753097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:55.754757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:55.755210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:55.755648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:55.756104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58876577-334c-4bec-a679-a88730fc3104 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:55.756725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:55.757250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:55.759369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:55.789377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 159/329] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:55.795667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:55.804251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:55.804877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:55.805168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:55.805458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:55.806206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:55.806503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:55.808022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:55.819304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 171/330] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:55.825112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:55.889199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:55.947320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c872d584-2ebd-41f3-8191-f930cba811d9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:55.972529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e8f491-4d4f-4de6-a2ad-170478c4840c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:56.003671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:56.004987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:56.005743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:56.006416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] PATH_INFO: `/v3/users/109ba80a80b74e48a8c17505d4be49ce` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:56.008575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] RBAC: Authorizing `identity:get_user(user_id=109ba80a80b74e48a8c17505d4be49ce)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:56.011600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:56.015488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 160/331] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:55 2022] GET /identity/v3/users/109ba80a80b74e48a8c17505d4be49ce => generated 293 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:56.028505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:56.030147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:56.031531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:56.032330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:56.033078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] PATH_INFO: `/v3/users/01b5dc097a31474895169eda2fcda733` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:56.036410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] RBAC: Authorizing `identity:get_user(user_id=01b5dc097a31474895169eda2fcda733)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:56.039611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:56.043724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 172/332] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:07:55 2022] GET /identity/v3/users/01b5dc097a31474895169eda2fcda733 => generated 293 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:56.053279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:56.121908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:56.224735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346dd01-5d0e-46cb-a29c-ac27d1fd4fd2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:56.251125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2c4d79-ecc3-4459-9c11-5a9884d6232c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:56.263863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:56.291479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ba6ffd-c733-4d62-b9cd-cace857bed91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:56.292821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba6ffd-c733-4d62-b9cd-cace857bed91 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:56.293299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba6ffd-c733-4d62-b9cd-cace857bed91 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:56.293722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba6ffd-c733-4d62-b9cd-cace857bed91 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:56.294899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba6ffd-c733-4d62-b9cd-cace857bed91 None admin] RBAC: Adding query filter params (role.id=51f5bbe85ea14c879a8c35831e619357, user.id=109ba80a80b74e48a8c17505d4be49ce, scope.system=all) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:56.295370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba6ffd-c733-4d62-b9cd-cace857bed91 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:56.297181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba6ffd-c733-4d62-b9cd-cace857bed91 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:56.319046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a36e70-48bc-443c-b4fe-1d4d00c8488e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:56.320322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a36e70-48bc-443c-b4fe-1d4d00c8488e None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:56.321254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a36e70-48bc-443c-b4fe-1d4d00c8488e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:56.322114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a36e70-48bc-443c-b4fe-1d4d00c8488e None admin] PATH_INFO: `/v3/system/users/01b5dc097a31474895169eda2fcda733/roles/b19c6f50e2f542aa81ec4cf175cba448` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:56.325320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 161/333] 10.222.0.52 () {56 vars in 1308 bytes} [Sat Aug 27 02:07:56 2022] GET /identity/v3/role_assignments?user.id=109ba80a80b74e48a8c17505d4be49ce&scope.system=all&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 510 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:07:56.325553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a36e70-48bc-443c-b4fe-1d4d00c8488e None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=01b5dc097a31474895169eda2fcda733, role_id=b19c6f50e2f542aa81ec4cf175cba448)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:56.329410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a36e70-48bc-443c-b4fe-1d4d00c8488e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:56.345998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 173/334] 10.222.0.52 () {58 vars in 1334 bytes} [Sat Aug 27 02:07:56 2022] PUT /identity/v3/system/users/01b5dc097a31474895169eda2fcda733/roles/b19c6f50e2f542aa81ec4cf175cba448 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:07:58.075703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b764038-99fd-4769-b2e6-d0fc75804f13 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:58.076487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b764038-99fd-4769-b2e6-d0fc75804f13 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:58.076487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b764038-99fd-4769-b2e6-d0fc75804f13 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:58.078337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 162/335] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:58.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fb9070-c3d8-4a8c-9a0b-0b10995ee41f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:58.094338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fb9070-c3d8-4a8c-9a0b-0b10995ee41f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:58.094338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fb9070-c3d8-4a8c-9a0b-0b10995ee41f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:58.131049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a1fb9070-c3d8-4a8c-9a0b-0b10995ee41f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:58.133228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a1fb9070-c3d8-4a8c-9a0b-0b10995ee41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:58.232754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 174/336] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:58.629605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2afa5-e4d3-4b43-b142-ea7a4a932faa None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:58.630285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2afa5-e4d3-4b43-b142-ea7a4a932faa None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:58.630822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2afa5-e4d3-4b43-b142-ea7a4a932faa None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:58.633406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 163/337] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:07:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:07:58.654280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a2f36-3c5a-4bb9-9547-8bb7d20d387f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:58.654716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a2f36-3c5a-4bb9-9547-8bb7d20d387f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:58.654871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a2f36-3c5a-4bb9-9547-8bb7d20d387f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:58.694884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-556a2f36-3c5a-4bb9-9547-8bb7d20d387f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:07:58.696753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-556a2f36-3c5a-4bb9-9547-8bb7d20d387f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:07:58.806158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 175/338] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:07:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:07:59.072916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a2afa5-e4d3-4b43-b142-ea7a4a932faa None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:59.255261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a2afa5-e4d3-4b43-b142-ea7a4a932faa None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:59.269382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:59.321500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:59.322161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:59.322440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:59.322709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:59.339431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:59.344923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:59.346829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:07:59.348505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:59.353181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 164/339] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:07:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:07:59.353978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556a2f36-3c5a-4bb9-9547-8bb7d20d387f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:59.407519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:59.505748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556a2f36-3c5a-4bb9-9547-8bb7d20d387f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:59.525358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:59.575992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:59.576853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:59.577265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:59.577661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:59.578805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:07:59.579231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:07:59.582218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:07:59.598510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 176/340] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:07:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:07:59.610347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:59.799762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63600d14-4da6-43f8-9b5f-1e5a6efaa3ab None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:59.858020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:07:59.862737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:07:59.864314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:07:59.864763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:07:59.867545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:07:59.879625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c94dea-129c-4b4a-b39f-d966014ea938 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:07:59.901711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:07:59.914870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:07:59.917578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:07:59.919669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 165/341] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:07:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 395 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:07:59.935477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:07:59.984901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:00.015832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c1708-7be4-4b4f-866f-a8f8a995bd8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:00.016986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c1708-7be4-4b4f-866f-a8f8a995bd8c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.017515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c1708-7be4-4b4f-866f-a8f8a995bd8c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.017980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c1708-7be4-4b4f-866f-a8f8a995bd8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.093518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1eedcf-2d1e-4974-a8f6-aae88b8dcb72 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:00.186552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:00.187138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.187607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.188081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.189096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:00.189427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:00.193165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:00.235428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 166/342] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:07:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:00.240465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c1708-7be4-4b4f-866f-a8f8a995bd8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:00.242754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c1708-7be4-4b4f-866f-a8f8a995bd8c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:00.243834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:00.416329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:00.426168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 177/343] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:07:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 817 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:00.428171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c011f-064c-411e-80da-f2ea94d7dec7 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.428636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c011f-064c-411e-80da-f2ea94d7dec7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.429074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c011f-064c-411e-80da-f2ea94d7dec7 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.432098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 178/344] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:00.434005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d536dcc-2344-4c36-9f5c-765c99fe413e None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.434450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d536dcc-2344-4c36-9f5c-765c99fe413e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.434947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d536dcc-2344-4c36-9f5c-765c99fe413e None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.436947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 179/345] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:00.442692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9c736-2902-426e-a4be-33d2b271be29 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.443140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9c736-2902-426e-a4be-33d2b271be29 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.443687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9c736-2902-426e-a4be-33d2b271be29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.467147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:00.473680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-28f9c736-2902-426e-a4be-33d2b271be29 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:00.475915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-28f9c736-2902-426e-a4be-33d2b271be29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:00.490271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b764aa84-308a-4fa7-8a6c-85db8f255602 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:00.550149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:00.550782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.551062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.551334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.616386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 180/346] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:00.617998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f9c736-2902-426e-a4be-33d2b271be29 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:00.700114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:00.702761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:00.858614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f9c736-2902-426e-a4be-33d2b271be29 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:00.897674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:00.920154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c662a5fd-fbc8-441a-916b-f7529114cf44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:00.921419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662a5fd-fbc8-441a-916b-f7529114cf44 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.921898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662a5fd-fbc8-441a-916b-f7529114cf44 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.922377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662a5fd-fbc8-441a-916b-f7529114cf44 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.924484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c662a5fd-fbc8-441a-916b-f7529114cf44 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:00.927853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c662a5fd-fbc8-441a-916b-f7529114cf44 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:00.931034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 181/347] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:00 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:00.932572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8ad370-a0d0-4ca3-a9e9-ebdd27582df9 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:00.933010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8ad370-a0d0-4ca3-a9e9-ebdd27582df9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:00.933518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8ad370-a0d0-4ca3-a9e9-ebdd27582df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:00.954554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:00.973383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ba8ad370-a0d0-4ca3-a9e9-ebdd27582df9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:00.974629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ba8ad370-a0d0-4ca3-a9e9-ebdd27582df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:00.994273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 167/348] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 752 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 27 02:08:00.995458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:01.211521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 182/349] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:01.212800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8ad370-a0d0-4ca3-a9e9-ebdd27582df9 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:01.337526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0d82c7-6722-4883-9735-111ddcf5140e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:01.409444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:01.410922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:01.411544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:01.412150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:01.421174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8ad370-a0d0-4ca3-a9e9-ebdd27582df9 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:01.422077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:01.426653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:01.429906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 168/350] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:00 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 435 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:01.431554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:01.459933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:01.481896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be43fa-4650-4285-9058-a426f974eaea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:01.483453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be43fa-4650-4285-9058-a426f974eaea None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:01.484089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be43fa-4650-4285-9058-a426f974eaea None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:01.484682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be43fa-4650-4285-9058-a426f974eaea None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:01.487022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4be43fa-4650-4285-9058-a426f974eaea None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:01.490073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4be43fa-4650-4285-9058-a426f974eaea None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:01.493510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 183/351] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:01 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:01.495933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6a4d9-1ef4-49d5-afa6-ed6c91cdd083 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:01.496547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6a4d9-1ef4-49d5-afa6-ed6c91cdd083 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:01.497015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f6a4d9-1ef4-49d5-afa6-ed6c91cdd083 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:01.499634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 184/352] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:01.501360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f6a4d9-1ef4-49d5-afa6-ed6c91cdd083 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:01.545713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:01.655029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fc80b2-b394-4134-804d-d36ce8c95484 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:01.658736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f6a4d9-1ef4-49d5-afa6-ed6c91cdd083 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:01.718046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c0a728-b6e7-4772-81e1-dddcfb4c6ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:01.718403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c0a728-b6e7-4772-81e1-dddcfb4c6ab3 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:01.719667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c0a728-b6e7-4772-81e1-dddcfb4c6ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:01.719667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c0a728-b6e7-4772-81e1-dddcfb4c6ab3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:01.720188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c0a728-b6e7-4772-81e1-dddcfb4c6ab3 None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=abd8cdfe3c4b488eb590b64706ada806) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:01.720729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c0a728-b6e7-4772-81e1-dddcfb4c6ab3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:01.722916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:01.723668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c0a728-b6e7-4772-81e1-dddcfb4c6ab3 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:01.724319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:01.724730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:01.725198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:01.732993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 185/353] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:01 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 260 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:01.734738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aca8ec-893b-4632-81ab-53c185873ffa None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:01.735150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aca8ec-893b-4632-81ab-53c185873ffa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:01.735606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aca8ec-893b-4632-81ab-53c185873ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:01.744955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:01.754774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:01.756321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:01.758031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 169/354] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:01.758986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:01.767027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-28aca8ec-893b-4632-81ab-53c185873ffa None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:01.770017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-28aca8ec-893b-4632-81ab-53c185873ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:01.868177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:01.912935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 186/355] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:01.912935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27a99e-b255-4252-b95f-b3b59cad1f75 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:01.912935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27a99e-b255-4252-b95f-b3b59cad1f75 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:01.912935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27a99e-b255-4252-b95f-b3b59cad1f75 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:01.912935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 187/356] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:01.914461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27a99e-b255-4252-b95f-b3b59cad1f75 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:01.978395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78083dc-8e36-4b6f-ad6f-1c09968fc96b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:02.016247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:02.077007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:02.078761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:02.079477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:02.080473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:02.083455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:02.086726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:02.089385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 170/357] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:01 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:02.090219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:02.154401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27a99e-b255-4252-b95f-b3b59cad1f75 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:02.225287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36551e3f-5e58-4865-adc7-d7e4693050ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:02.226001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36551e3f-5e58-4865-adc7-d7e4693050ef None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:02.226285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36551e3f-5e58-4865-adc7-d7e4693050ef None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:02.226555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36551e3f-5e58-4865-adc7-d7e4693050ef None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:02.242745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36551e3f-5e58-4865-adc7-d7e4693050ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-36551e3f-5e58-4865-adc7-d7e4693050ef None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:02.250857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:02.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 27 02:08:02.252877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:02.254024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 188/358] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:02.255007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72bca2b-155c-431a-8097-ba820bf8516d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:02.255143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72bca2b-155c-431a-8097-ba820bf8516d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:02.255389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72bca2b-155c-431a-8097-ba820bf8516d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:02.292040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c419a4-747c-4fa8-887c-c99104050ec2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:02.302694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f72bca2b-155c-431a-8097-ba820bf8516d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:02.306178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f72bca2b-155c-431a-8097-ba820bf8516d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:02.372244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:02.374798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:02.377216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:02.378144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:02.379029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:02.379464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:02.381084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:02.402153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 171/359] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:08:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:02.402342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:02.403578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:02.471435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 189/360] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:02.471435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72bca2b-155c-431a-8097-ba820bf8516d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:02.520274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:02.633163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f874cafa-3f77-46e7-b868-7fc4511733d2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:02.645764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72bca2b-155c-431a-8097-ba820bf8516d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:02.689462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:02.690298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:02.690503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:02.690780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:02.692058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:02.692445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:02.694462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:02.696484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:02.697843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:02.698372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:02.698847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:02.700041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=abd8cdfe3c4b488eb590b64706ada806) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:02.700677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:02.703033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:02.710407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 190/361] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:08:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:02.711638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:02.713708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 172/362] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:02 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 260 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:02.715244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:02.761414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:02.892001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27ede93-7855-45ab-9606-d0979e7d9420 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:02.929195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81bfe4a-2086-4eda-b61d-3c47a0e60bf7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:02.953434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:02.958267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:02.959907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:02.960588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:02.961063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:02.983280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:02.994450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:02.995574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:02.996373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:02.997060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 191/363] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:08:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:02.998053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:03.004820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:03.006442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:03.007182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:03.008005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:03.151548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:03.154346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:03.215848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:03.229710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab6b4a6-b7e8-4636-9004-95bd4de64bd6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:03.300662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:03.303103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:03.303876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:03.304580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] PATH_INFO: `/v3/users/01b5dc097a31474895169eda2fcda733` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:03.307395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] RBAC: Authorizing `identity:get_user(user_id=01b5dc097a31474895169eda2fcda733)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:03.310912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:03.314906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 192/364] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:08:02 2022] GET /identity/v3/users/01b5dc097a31474895169eda2fcda733 => generated 293 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:03.317404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:03.385671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 173/365] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 671 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:03.387041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:03.413832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:03.564245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b58303b-4326-4318-b905-ef6d82453892 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:03.625829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:03.626847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:03.627451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:03.627786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7c0265-761f-4c7c-a2cd-350e04cd708e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:03.627922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:03.629082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:03.629656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:03.631722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:03.641734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 193/366] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:03.643149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:03.694640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:03.709109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:03.717891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:03.718331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:03.718766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:03.740926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:03.750384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:03.753722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:03.756740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 174/367] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:08:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 366 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:03.767262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:03.822410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabfdd87-4c90-439d-8511-967aa3504428 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:03.881196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:03.881907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:03.882284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:03.882585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:03.883674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] RBAC: Adding query filter params (role.id=b19c6f50e2f542aa81ec4cf175cba448, user.id=01b5dc097a31474895169eda2fcda733, scope.system=all) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:03.884053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:03.886076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:03.894044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:03.902402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 194/368] 10.222.0.52 () {56 vars in 1308 bytes} [Sat Aug 27 02:08:03 2022] GET /identity/v3/role_assignments?user.id=01b5dc097a31474895169eda2fcda733&scope.system=all&role.id=b19c6f50e2f542aa81ec4cf175cba448 => generated 510 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:03.903283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:03.968716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeda14a-7451-4a31-8e64-585f33f45955 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:04.027873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:04.036820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:04.038022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:04.038491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:04.038954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:04.041248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:04.044405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:04.047616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 175/369] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:03 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:04.049572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:04.061223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99b49c-a4bd-473b-9915-dd578ca5d3b0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:04.108151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:04.109415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:04.109984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:04.110601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:04.210920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:04.212562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:04.277676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e5babe-1074-4539-85b9-73991379d1a0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:04.321716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:04.345061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 195/370] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:04.346162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:04.346712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:04.347983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:04.348586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:04.349059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:04.350444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:04.350975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:04.353576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:04.359648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:04.374683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 176/371] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:04.376151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:04.479793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3b9c63-4c4d-4ed9-a49c-2d11eb090cb0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:04.528115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:04.528819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:04.529310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:04.529576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:04.530793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:04.532262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:04.534677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 196/372] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:04 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:04.535742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:04.585062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50813b19-e996-434c-89b5-84ea4ec83bc3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:04.625322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:04.665464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:04.666677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:04.667141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:04.667601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:04.669427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:04.670998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:04.674923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 177/373] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:04 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:04.676159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:04.708832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c0172-28ad-411c-84c6-26409e346cbe None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:04.791288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:04.793380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:04.793380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:04.793866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:04.853710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:04.941593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9c4af-f1a6-4b8a-a537-6844d7b95de7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:04.992964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:04.995021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:05.017311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:05.018328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:05.018838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:05.019393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:05.020931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=64d8a5429b86463da1df922aeb084e5f) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:05.021515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:05.024329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:05.035378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 178/374] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:04 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 260 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:05.036996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:05.086081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:05.174969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 197/375] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 639 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:05.185361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:05.185592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc5d32-7227-4023-a6ea-8556d918a359 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:05.228444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:05.238248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:05.239194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:05.239607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:05.240039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:05.241574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:05.243149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:05.246446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 179/376] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:05 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:05.255007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:05.473293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b369ac28-7124-44c5-ad2c-60a31fce11e5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:05.490351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a789b7bb-7e2b-4685-8699-5be9d775ede7 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:05.540968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:05.542054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:05.542614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:05.543026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:05.543946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=64d8a5429b86463da1df922aeb084e5f) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:05.544405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:05.547071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:05.555903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 180/377] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:05 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 260 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:05.567375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6267e9f-13d5-4432-a136-0cabb314f6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:05.568484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6267e9f-13d5-4432-a136-0cabb314f6ca None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:05.569030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6267e9f-13d5-4432-a136-0cabb314f6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:05.569556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6267e9f-13d5-4432-a136-0cabb314f6ca None admin] PATH_INFO: `/v3/projects/f1df9f889b3642a7ac056884bc2c51a4` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:05.571731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6267e9f-13d5-4432-a136-0cabb314f6ca None admin] RBAC: Authorizing `identity:get_project(project_id=f1df9f889b3642a7ac056884bc2c51a4)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:05.574534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6267e9f-13d5-4432-a136-0cabb314f6ca None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:05.577579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 198/378] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:05 2022] GET /identity/v3/projects/f1df9f889b3642a7ac056884bc2c51a4 => generated 344 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:05.586336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:05.630219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef45914-80a1-4639-bba2-4eafa74f72e5 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:05.630804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef45914-80a1-4639-bba2-4eafa74f72e5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:05.631648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef45914-80a1-4639-bba2-4eafa74f72e5 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:05.635255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 199/379] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:05.654157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b965be-dd30-42ee-9fde-3c94120ed0d5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:05.655204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b965be-dd30-42ee-9fde-3c94120ed0d5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:05.655547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b965be-dd30-42ee-9fde-3c94120ed0d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:05.688360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:05.703469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a9b965be-dd30-42ee-9fde-3c94120ed0d5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:05.705590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a9b965be-dd30-42ee-9fde-3c94120ed0d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:05.737265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:05.796979 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a36d0f-9652-4bbd-b701-4c4d44e2c04f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:05.855962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:05.862152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:05.862882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:05.863444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] PATH_INFO: `/v3/groups/7f5d7675d60e4a0eaed89a8e62532d89` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:05.872032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] RBAC: Authorizing `identity:get_group(group_id=7f5d7675d60e4a0eaed89a8e62532d89)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:05.873009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 200/380] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:05.874038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:05.875907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 181/381] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:05 2022] GET /identity/v3/groups/7f5d7675d60e4a0eaed89a8e62532d89 => generated 229 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:05.886024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b965be-dd30-42ee-9fde-3c94120ed0d5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:06.073339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b965be-dd30-42ee-9fde-3c94120ed0d5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:06.136845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:06.137994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:06.138435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:06.138840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:06.140124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] RBAC: Adding query filter params (group.id=7f5d7675d60e4a0eaed89a8e62532d89, role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=f1df9f889b3642a7ac056884bc2c51a4) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:06.140870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:06.143324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:06.152032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 201/382] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:05 2022] GET /identity/v3/role_assignments?group.id=7f5d7675d60e4a0eaed89a8e62532d89&scope.project.id=f1df9f889b3642a7ac056884bc2c51a4&role.id=b83e461b8f6643fba45f427d28498c41 => generated 260 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:06.541514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:06.761153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccdb0dc-435a-4856-a44b-0841bf8e3e21 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:06.810332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:06.831675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027e6360-e75d-4f1e-8672-13f0db117b70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:06.832654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027e6360-e75d-4f1e-8672-13f0db117b70 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:06.833080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027e6360-e75d-4f1e-8672-13f0db117b70 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:06.833541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027e6360-e75d-4f1e-8672-13f0db117b70 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:06.852545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027e6360-e75d-4f1e-8672-13f0db117b70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-027e6360-e75d-4f1e-8672-13f0db117b70 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:06.862420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:06.864630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:06.866213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 182/383] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:06.873335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:06.883832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89953c23-57d0-4b84-ac8d-cf66e2d069e9 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:06.884304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89953c23-57d0-4b84-ac8d-cf66e2d069e9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:06.885703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89953c23-57d0-4b84-ac8d-cf66e2d069e9 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:06.887525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 183/384] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:08:06.898931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3140e7-11bb-416b-a80a-1ac42d248204 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:06.899268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3140e7-11bb-416b-a80a-1ac42d248204 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:06.899520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3140e7-11bb-416b-a80a-1ac42d248204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:06.917718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:06.922445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1c3140e7-11bb-416b-a80a-1ac42d248204 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:06.923669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1c3140e7-11bb-416b-a80a-1ac42d248204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:07.102204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 184/385] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:07.168079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc89d91-f0ee-4646-9bda-2a8e75fc0a60 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:07.240624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:07.241926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:07.242621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:07.243218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:07.244504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:07.245190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:07.247857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:07.272391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 202/386] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:08:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 399 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:07.282491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3140e7-11bb-416b-a80a-1ac42d248204 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:07.483585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3140e7-11bb-416b-a80a-1ac42d248204 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:07.524099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:07.556649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:07.557337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:07.557613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:07.557879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:07.559107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:07.560611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:07.562622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 185/387] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:07 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:07.571773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:07.650546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:07.760407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:07.798428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2be9c5-5449-4c9c-ba99-6d4fdeb08434 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:07.847511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:07.848165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:07.848513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:07.848787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:07.850166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:07.851618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:07.853694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 203/388] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:07 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:07.870311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:07.931334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e3bc54-46ae-4ed7-a34d-99458aebad9f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:08.035727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:08.036456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:08.036811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:08.037160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:08.062639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:08.069013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:08.070905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:08.072449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 186/389] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:08:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 420 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:08.091429 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:08.168141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3d31fa-7217-445a-bc3f-a47da8fcf977 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:08.196654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:08.271845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:08.272914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:08.273339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:08.273791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806/groups/1848218daa7a489394e0133a4f8d46b0/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:08.283423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] RBAC: Authorizing `identity:create_grant(project_id=abd8cdfe3c4b488eb590b64706ada806, group_id=1848218daa7a489394e0133a4f8d46b0, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:08.286266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:08.289771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc025e48-77dc-439e-a0ad-481028bb6b7c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:08.320230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 204/390] 10.222.0.52 () {58 vars in 1442 bytes} [Sat Aug 27 02:08:07 2022] PUT /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806/groups/1848218daa7a489394e0133a4f8d46b0/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:08.337951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:08.339170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:08.339986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:08.340863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:08.342621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:08.343326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:08.345011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:08.360202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 187/391] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:08.366041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:08.373661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:08.542508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9821c87d-a724-4cb4-991b-84b0d8f58a3a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:08.601391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30f3f2d-a2b0-4cce-9b78-c79bc3873e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:08.602180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f3f2d-a2b0-4cce-9b78-c79bc3873e82 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:08.602589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f3f2d-a2b0-4cce-9b78-c79bc3873e82 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:08.602931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f3f2d-a2b0-4cce-9b78-c79bc3873e82 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:08.605738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30f3f2d-a2b0-4cce-9b78-c79bc3873e82 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:08.608135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30f3f2d-a2b0-4cce-9b78-c79bc3873e82 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:08.611540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 205/392] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:08 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:08.618052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:08.753625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249cc523-d1bf-4bf7-ae94-bd1c76b747e6 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:08.754636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249cc523-d1bf-4bf7-ae94-bd1c76b747e6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:08.755330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249cc523-d1bf-4bf7-ae94-bd1c76b747e6 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:08.759579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 206/393] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:08 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:08:08.768860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b30807f-824c-4118-857a-017c07684533 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:08.769718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b30807f-824c-4118-857a-017c07684533 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:08.770352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b30807f-824c-4118-857a-017c07684533 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:08.803042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-9b30807f-824c-4118-857a-017c07684533 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:08.805618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9b30807f-824c-4118-857a-017c07684533 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:08.878007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c8513-e8a5-4c04-887d-1c5d4eff0b8c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:08.902503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:08.937041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 207/394] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:08.987393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f450925-fa85-4573-b3b5-833ae1669862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:08.988413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f450925-fa85-4573-b3b5-833ae1669862 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:08.988846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f450925-fa85-4573-b3b5-833ae1669862 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:08.989337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f450925-fa85-4573-b3b5-833ae1669862 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:08.994374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f450925-fa85-4573-b3b5-833ae1669862 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:08.997066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f450925-fa85-4573-b3b5-833ae1669862 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:09.000457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 188/395] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:08 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 383 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:09.009890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b30807f-824c-4118-857a-017c07684533 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:09.014810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:09.166725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8640ed9d-27d2-4473-a15e-a533efca88cd None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:09.166983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8640ed9d-27d2-4473-a15e-a533efca88cd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:09.167337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8640ed9d-27d2-4473-a15e-a533efca88cd None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:09.168626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 189/396] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:08:09.175798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63eb845-c756-4dc5-8942-d345eb347faa None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:09.176189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63eb845-c756-4dc5-8942-d345eb347faa None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:09.176421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63eb845-c756-4dc5-8942-d345eb347faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:09.198704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c63eb845-c756-4dc5-8942-d345eb347faa None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:09.199953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c63eb845-c756-4dc5-8942-d345eb347faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:09.210834 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b30807f-824c-4118-857a-017c07684533 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:09.287510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:09.288198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:09.288471 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:09.288727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806/groups/1848218daa7a489394e0133a4f8d46b0/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:09.290975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] RBAC: Authorizing `identity:create_grant(project_id=abd8cdfe3c4b488eb590b64706ada806, group_id=1848218daa7a489394e0133a4f8d46b0, role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:09.292599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:09.295464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 190/397] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:09.297036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e9da7c-d51a-4890-b687-8d91347d7e9a None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:09.297663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e9da7c-d51a-4890-b687-8d91347d7e9a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:09.305300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e9da7c-d51a-4890-b687-8d91347d7e9a None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:09.305300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 191/398] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:09.305704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d926a4bd-c637-4289-9263-bbd76706194a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:09.305993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d926a4bd-c637-4289-9263-bbd76706194a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:09.306359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d926a4bd-c637-4289-9263-bbd76706194a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:09.318768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 208/399] 10.222.0.52 () {58 vars in 1442 bytes} [Sat Aug 27 02:08:09 2022] PUT /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806/groups/1848218daa7a489394e0133a4f8d46b0/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:09.329407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-d926a4bd-c637-4289-9263-bbd76706194a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:09.331150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d926a4bd-c637-4289-9263-bbd76706194a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:09.393443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:09.442240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 192/400] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:09.558062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:09.625733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911446c7-79d1-46da-a451-4348d871fcea None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:09.696677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:09.697277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:09.697556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:09.697829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:09.722097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:09.727923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:09.729722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:09.731165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 209/401] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:09.738629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d926a4bd-c637-4289-9263-bbd76706194a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:09.796792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:09.822460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:09.899957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d926a4bd-c637-4289-9263-bbd76706194a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:09.961314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:09.961831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:09.962107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:09.962373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:09.963893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] RBAC: Adding query filter params (name=member) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:09.965970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:09.967640 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:09.981790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 193/402] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:08:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:09.982699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.018096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0e032-f770-4168-b0a8-1a6ad7aa4885 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:10.068914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:10.095291 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:10.095995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:10.096307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:10.096777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:10.111370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:10.118710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:10.119862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:10.120956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:10.125583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 210/403] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:08:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:10.126675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.158037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:10.168775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0eff3d-f2e1-41aa-8f6c-8e57e93e0e77 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:10.223011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:10.223563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:10.223847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:10.224236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:10.242048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:10.248083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:10.249440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 194/404] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:08:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:10.254118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.295239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f21dc9f-2d45-48eb-bcee-8d9fe0cc6981 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:10.348896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:10.349975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:10.350417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:10.350921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:10.353024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:10.355634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:10.363806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 211/405] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:10 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:10.363806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.427194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:10.429405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154660e9-8b8f-4f51-9af2-0dcfad22a790 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:10.477477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:10.478355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:10.478786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:10.479206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:10.480048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:10.480516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:10.482159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:10.495389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 195/406] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:10.497161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.506890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ca3df0-2806-4653-9033-c921d17cccb3 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:10.529605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:10.566997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:10.567973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:10.568384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:10.568818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:10.570003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:10.570458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:10.573650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:10.597157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 212/407] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:10.597157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.711164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a04093-cb3a-41b2-8fab-b0a76e018283 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:10.768356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:10.768848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:10.769168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:10.769533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:10.771103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:10.772695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:10.781153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 196/408] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:10 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:10.781153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.806795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b497c7-a32f-4381-83bf-22ac0acbcd90 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:10.832964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:10.877204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:10.878382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:10.878880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:10.879361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:10.880993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:10.882928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:10.885696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 213/409] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:10 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:10.887075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:10.953185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:10.976410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c41b39-4e8d-4c95-9bac-db3dd45f1bdc None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:11.026763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8717d5f5-fd46-47f3-a2ed-c095ec328a79 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:11.033026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:11.033844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.034362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.034838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] PATH_INFO: `/v3/groups/7f5d7675d60e4a0eaed89a8e62532d89` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.037002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] RBAC: Authorizing `identity:get_group(group_id=7f5d7675d60e4a0eaed89a8e62532d89)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:11.039915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:11.043244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 197/410] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:10 2022] GET /identity/v3/groups/7f5d7675d60e4a0eaed89a8e62532d89 => generated 229 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:11.050988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:11.083689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:11.085450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.086319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.087249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f/groups/1848218daa7a489394e0133a4f8d46b0/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.091442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] RBAC: Authorizing `identity:create_grant(project_id=64d8a5429b86463da1df922aeb084e5f, group_id=1848218daa7a489394e0133a4f8d46b0, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:11.093892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:11.123842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 214/411] 10.222.0.52 () {58 vars in 1442 bytes} [Sat Aug 27 02:08:10 2022] PUT /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f/groups/1848218daa7a489394e0133a4f8d46b0/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:11.125286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:11.232997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:11.327048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05157f8-0011-4e60-bdf8-758ec5505c10 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:11.419809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de9548f-e5a3-4e01-87c5-316fd0322e7b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:11.454216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:11.455785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.456118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.456397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.461101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:11.465494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:11.469318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 198/412] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:11 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 425 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:11.476485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:11.523591 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed93ccb-b403-44a6-9562-3ff4b68b2ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:11.525549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed93ccb-b403-44a6-9562-3ff4b68b2ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.526034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed93ccb-b403-44a6-9562-3ff4b68b2ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.526721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed93ccb-b403-44a6-9562-3ff4b68b2ec9 None admin] PATH_INFO: `/v3/projects/f1df9f889b3642a7ac056884bc2c51a4` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.533575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed93ccb-b403-44a6-9562-3ff4b68b2ec9 None admin] RBAC: Authorizing `identity:get_project(project_id=f1df9f889b3642a7ac056884bc2c51a4)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:11.533575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed93ccb-b403-44a6-9562-3ff4b68b2ec9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:11.535397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 215/413] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:11 2022] GET /identity/v3/projects/f1df9f889b3642a7ac056884bc2c51a4 => generated 344 bytes in 411 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:11.537983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f321db0-5e21-4b4d-9969-1ee394f70cd4 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.538469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f321db0-5e21-4b4d-9969-1ee394f70cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.539007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f321db0-5e21-4b4d-9969-1ee394f70cd4 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.541686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:11.542286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 216/414] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:11.543950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f321db0-5e21-4b4d-9969-1ee394f70cd4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:11.702184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d5f661-ca58-4c9a-9c90-f5d7b731e6e1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:11.756223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e8efeb-76f5-403e-a416-6bdc4a972194 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:11.757070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8efeb-76f5-403e-a416-6bdc4a972194 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.757483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8efeb-76f5-403e-a416-6bdc4a972194 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.757951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8efeb-76f5-403e-a416-6bdc4a972194 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f/groups/1848218daa7a489394e0133a4f8d46b0/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.761613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e8efeb-76f5-403e-a416-6bdc4a972194 None admin] RBAC: Authorizing `identity:create_grant(project_id=64d8a5429b86463da1df922aeb084e5f, group_id=1848218daa7a489394e0133a4f8d46b0, role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:11.762308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f321db0-5e21-4b4d-9969-1ee394f70cd4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:11.763353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e8efeb-76f5-403e-a416-6bdc4a972194 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:11.780229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:11.789178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 199/415] 10.222.0.52 () {58 vars in 1442 bytes} [Sat Aug 27 02:08:11 2022] PUT /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f/groups/1848218daa7a489394e0133a4f8d46b0/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:08:11.789178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a9ea6f-be5c-4942-bf7f-5650c526bf7e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.789178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a9ea6f-be5c-4942-bf7f-5650c526bf7e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.789178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a9ea6f-be5c-4942-bf7f-5650c526bf7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.820355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b0a9ea6f-be5c-4942-bf7f-5650c526bf7e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:11.820355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b0a9ea6f-be5c-4942-bf7f-5650c526bf7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:11.862667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67dfa19-1438-41db-b6bf-6713ed1f17a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:11.865231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67dfa19-1438-41db-b6bf-6713ed1f17a2 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:11.865536 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67dfa19-1438-41db-b6bf-6713ed1f17a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:11.865782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67dfa19-1438-41db-b6bf-6713ed1f17a2 None admin] PATH_INFO: `/v3/projects/f1df9f889b3642a7ac056884bc2c51a4/groups/7f5d7675d60e4a0eaed89a8e62532d89/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:11.873152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67dfa19-1438-41db-b6bf-6713ed1f17a2 None admin] RBAC: Authorizing `identity:create_grant(project_id=f1df9f889b3642a7ac056884bc2c51a4, group_id=7f5d7675d60e4a0eaed89a8e62532d89, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:11.874718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67dfa19-1438-41db-b6bf-6713ed1f17a2 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:11.909075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 217/416] 10.222.0.52 () {58 vars in 1442 bytes} [Sat Aug 27 02:08:11 2022] PUT /identity/v3/projects/f1df9f889b3642a7ac056884bc2c51a4/groups/7f5d7675d60e4a0eaed89a8e62532d89/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:11.977231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 200/417] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:12.300624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0f2bbe-d61b-442f-a7a7-853bf85e549f None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:12.301164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0f2bbe-d61b-442f-a7a7-853bf85e549f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:12.301483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0f2bbe-d61b-442f-a7a7-853bf85e549f None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:12.303923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 218/418] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:12.310832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0749f5-0cdf-4842-bd19-47b37ab633a7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:12.311205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0749f5-0cdf-4842-bd19-47b37ab633a7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:12.311459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0749f5-0cdf-4842-bd19-47b37ab633a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:12.341832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-db0749f5-0cdf-4842-bd19-47b37ab633a7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:12.343582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-db0749f5-0cdf-4842-bd19-47b37ab633a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:12.358611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:12.408454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0f2bbe-d61b-442f-a7a7-853bf85e549f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:12.508454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 201/419] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:12.621079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0f2bbe-d61b-442f-a7a7-853bf85e549f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:12.693696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:12.694815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:12.695323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:12.695854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:12.715000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:12.723541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:12.725195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:12.726631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:12.736407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 219/420] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:08:12.740288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0749f5-0cdf-4842-bd19-47b37ab633a7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:13.010760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:13.034119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0749f5-0cdf-4842-bd19-47b37ab633a7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:13.073275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:13.127160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:13.129022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:13.129878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:13.130584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:13.132460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] RBAC: Adding query filter params (name=member) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:13.133582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:13.136679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:13.157441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 202/421] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:08:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 418 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:13.165348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:13.180048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:13.245476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae36033-46ac-4d13-aa64-eefc06f9b2e6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:13.325714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:13.327927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:13.328747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:13.329657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:13.367672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:13.388644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:13.390394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:13.392313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:13.402508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 220/422] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:08:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 391 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:08:13.411178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:13.475688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411137d8-7886-4fdc-809e-02c1a675d4eb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:13.567787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:13.568793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:13.569305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:13.569757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:13.664372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:13.732605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7c9106-5b9b-4201-be13-2b6ed40912e3 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:13.747715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:13.751640 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:13.757599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:13.834741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:13.835802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:13.836263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:13.836708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:13.837967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:13.838493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:13.841199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:13.869161 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 221/423] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 459 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:13.878229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:13.969974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 203/424] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 805 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:13.980513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:14.271994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9d9c7b-b61d-45af-ba26-389651cc43af None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:14.278579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d5e9ce-2839-480c-a289-f3f781db0342 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:14.286136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:14.388254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:14.389392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:14.389878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:14.390362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:14.392527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:14.394359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38646800-db2a-4302-9894-8d27ef773373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:14.395298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38646800-db2a-4302-9894-8d27ef773373 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:14.395450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:14.396075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38646800-db2a-4302-9894-8d27ef773373 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:14.396556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38646800-db2a-4302-9894-8d27ef773373 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:14.399059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 204/425] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:13 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 420 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:14.406367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:14.470055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:14.576054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38646800-db2a-4302-9894-8d27ef773373 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:14.578556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38646800-db2a-4302-9894-8d27ef773373 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:14.613661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5275db0-fac2-4268-812f-1d70fec6c509 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:14.672020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c285854-3e3b-4ca9-9ace-5d85c293ed96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:14.673312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c285854-3e3b-4ca9-9ace-5d85c293ed96 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:14.673791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c285854-3e3b-4ca9-9ace-5d85c293ed96 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:14.674277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c285854-3e3b-4ca9-9ace-5d85c293ed96 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:14.676486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c285854-3e3b-4ca9-9ace-5d85c293ed96 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:14.679311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c285854-3e3b-4ca9-9ace-5d85c293ed96 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:14.682996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 205/426] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:14 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:14.685271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4731ea4f-dd8b-4311-b0be-73b152420d0a None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:14.685975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4731ea4f-dd8b-4311-b0be-73b152420d0a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:14.686665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4731ea4f-dd8b-4311-b0be-73b152420d0a None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:14.691795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 206/427] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:14.693756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4731ea4f-dd8b-4311-b0be-73b152420d0a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:14.747755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 222/428] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 871 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 27 02:08:14.749446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e73928-2d51-4b1b-aa2b-c4fedffb6b5f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:14.749776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e73928-2d51-4b1b-aa2b-c4fedffb6b5f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:14.750154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e73928-2d51-4b1b-aa2b-c4fedffb6b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:14.762146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:14.782197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c0e73928-2d51-4b1b-aa2b-c4fedffb6b5f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:14.785433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c0e73928-2d51-4b1b-aa2b-c4fedffb6b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:14.852919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4731ea4f-dd8b-4311-b0be-73b152420d0a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:14.858485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:14.909521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 223/429] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:14.911009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e73928-2d51-4b1b-aa2b-c4fedffb6b5f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:14.912024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:14.912848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:14.913411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:14.913884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:14.915148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=abd8cdfe3c4b488eb590b64706ada806) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:14.915683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:14.917325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:14.926837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 207/430] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:14 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 611 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:15.106373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e73928-2d51-4b1b-aa2b-c4fedffb6b5f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:15.162763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807321b-697e-44da-9148-8af53a9cac4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:15.163557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807321b-697e-44da-9148-8af53a9cac4c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.163831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807321b-697e-44da-9148-8af53a9cac4c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.164103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807321b-697e-44da-9148-8af53a9cac4c None admin] PATH_INFO: `/v3/projects/abd8cdfe3c4b488eb590b64706ada806` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.165301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807321b-697e-44da-9148-8af53a9cac4c None admin] RBAC: Authorizing `identity:get_project(project_id=abd8cdfe3c4b488eb590b64706ada806)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:15.166833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807321b-697e-44da-9148-8af53a9cac4c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:15.168898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 224/431] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:14 2022] GET /identity/v3/projects/abd8cdfe3c4b488eb590b64706ada806 => generated 298 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:15.175195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:15.181375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda500d-a010-4175-ac81-cbef9042b906 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.181769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda500d-a010-4175-ac81-cbef9042b906 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.182227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda500d-a010-4175-ac81-cbef9042b906 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.184811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 225/432] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:15.203173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7267b2-19af-4c8a-af53-64134f41e309 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.203551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7267b2-19af-4c8a-af53-64134f41e309 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.203827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7267b2-19af-4c8a-af53-64134f41e309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.229276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5b7267b2-19af-4c8a-af53-64134f41e309 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:15.231971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5b7267b2-19af-4c8a-af53-64134f41e309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:15.321661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:15.358671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b82cf-b608-4ed5-ad38-c2292e93d2e4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:15.364136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:15.386014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 226/433] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:15.386014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7267b2-19af-4c8a-af53-64134f41e309 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:15.431089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:15.432482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.433031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.433633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.438164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:15.440325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:15.442514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 208/434] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:15 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:15.453552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:15.577234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7267b2-19af-4c8a-af53-64134f41e309 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:15.633035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4aa3c5-72a6-400f-b176-cd8281a96a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:15.633630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4aa3c5-72a6-400f-b176-cd8281a96a33 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.633907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4aa3c5-72a6-400f-b176-cd8281a96a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.634180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4aa3c5-72a6-400f-b176-cd8281a96a33 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.654238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4aa3c5-72a6-400f-b176-cd8281a96a33 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:15.654550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424f704a-e117-40f8-a7e5-9de446918da9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-be4aa3c5-72a6-400f-b176-cd8281a96a33 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:15.670208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:15.671521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 27 02:08:15.673544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:15.680546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 227/435] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:15.680546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15188c9-8567-480e-bb01-fe8c6231e2a8 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.680546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15188c9-8567-480e-bb01-fe8c6231e2a8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.680546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15188c9-8567-480e-bb01-fe8c6231e2a8 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.680546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 228/436] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:15.692736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15188c9-8567-480e-bb01-fe8c6231e2a8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:15.733463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6086820-f3b7-471c-a7a3-c15efcbd6290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:15.735145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6086820-f3b7-471c-a7a3-c15efcbd6290 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.735145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6086820-f3b7-471c-a7a3-c15efcbd6290 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.735971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6086820-f3b7-471c-a7a3-c15efcbd6290 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.738722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6086820-f3b7-471c-a7a3-c15efcbd6290 None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=abd8cdfe3c4b488eb590b64706ada806) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:15.738722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6086820-f3b7-471c-a7a3-c15efcbd6290 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:15.741580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6086820-f3b7-471c-a7a3-c15efcbd6290 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:15.757588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 209/437] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:15 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=abd8cdfe3c4b488eb590b64706ada806&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 611 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:15.760151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0fbf05-f80d-4416-9413-859c14701206 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.761080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0fbf05-f80d-4416-9413-859c14701206 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.762239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0fbf05-f80d-4416-9413-859c14701206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.779909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:15.814622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ac0fbf05-f80d-4416-9413-859c14701206 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:15.814622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ac0fbf05-f80d-4416-9413-859c14701206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:15.864322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:15.897097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15188c9-8567-480e-bb01-fe8c6231e2a8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:15.924620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 210/438] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:15.925301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0fbf05-f80d-4416-9413-859c14701206 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:15.965458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:15.967349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:15.968225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:15.968888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:15.970398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] RBAC: Adding query filter params (name=member) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:15.971137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:15.974512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:15.994871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 229/439] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:08:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:16.001861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:16.097471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0fbf05-f80d-4416-9413-859c14701206 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:16.158626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:16.159069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:16.159484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:16.159899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:16.165597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:16.170290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4e0b81-b45a-4c2c-ba0d-cfa0c146e593 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:16.180631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:16.187119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:16.188414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:16.189803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:16.192223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 211/440] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:08:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:16.199744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:16.225791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:16.226446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:16.226712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:16.226982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:16.252616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:16.323837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:16.325420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:16.361739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db927d2a-19ad-46b1-a273-317a90879b32 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:16.403484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:16.404383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:16.404739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:16.405088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:16.410382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:16.410740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:16.412462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:16.423337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 212/441] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:16.424715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:16.466376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:16.471857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 230/442] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:16.473447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:16.561344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:16.579874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68295980-2298-4331-88b3-7ee90d3a8ded None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:16.618910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1499430a-a346-423e-a028-5e1657c85c45 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:16.619147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:16.620087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:16.620557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:16.621032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:16.633960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:16.639866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:16.641772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 213/443] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:08:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:16.643670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:16.673543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:16.674537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:16.674537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:16.674537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:16.761377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:16.764112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:16.766067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:16.791715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478fb579-2c4d-42d3-a225-b277719f036b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:16.840162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:16.840814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:16.841401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:16.841597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:16.843143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:16.844926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:16.846885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 214/444] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:16 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:16.847935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:16.874526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:16.895962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 231/445] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:16.896863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.011952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84841f-08a1-4404-9d32-b9aff83fc3fd None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:17.025181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:17.029553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93deb5c-df1a-4d8c-8b00-5f1e810e200e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:17.061255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:17.062146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:17.062578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:17.062988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:17.064178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:17.064457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:17.066779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:17.072984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:17.074970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 215/446] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:17.075604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:17.075765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.076718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:17.077677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:17.079830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:17.081893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:17.084235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 232/447] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:16 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:17.085145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.201195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:17.216916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56106c96-f279-426a-a327-60f7d20fc21a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:17.231545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7cb87-cdc5-4239-ae52-decbb1fdfcff None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:17.274796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:17.275677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:17.276012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:17.276287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:17.278606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:17.279286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:17.279610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:17.279874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] PATH_INFO: `/v3/projects/64d8a5429b86463da1df922aeb084e5f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:17.281077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] RBAC: Authorizing `identity:get_project(project_id=64d8a5429b86463da1df922aeb084e5f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:17.282663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:17.284390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 216/448] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:17 2022] GET /identity/v3/projects/64d8a5429b86463da1df922aeb084e5f => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:17.285179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.326630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:17.372892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:17.375607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:17.424835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedab8e4-d043-4f66-aad6-c7c4c7637fc1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:17.476477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:17.477078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:17.477496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:17.477625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:17.478361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=51f5bbe85ea14c879a8c35831e619357, scope.project.id=64d8a5429b86463da1df922aeb084e5f) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:17.478667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:17.480171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:17.488455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 217/449] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:17 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=51f5bbe85ea14c879a8c35831e619357 => generated 611 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:17.489085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.507429 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:17.536531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 233/450] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:17.543312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.685239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374c59c-60a4-4334-b6b6-eea82f5014e1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:17.692776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:17.765765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:17.766641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:17.767060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:17.767342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] PATH_INFO: `/v3/groups/1848218daa7a489394e0133a4f8d46b0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:17.768918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] RBAC: Authorizing `identity:get_group(group_id=1848218daa7a489394e0133a4f8d46b0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:17.771195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:17.774458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 218/451] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:17 2022] GET /identity/v3/groups/1848218daa7a489394e0133a4f8d46b0 => generated 226 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:17.776747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ee9b50-c751-4a23-9885-8ce9f3955beb None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:17.794597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.856098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:17.857238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:17.857677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:17.858118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] PATH_INFO: `/v3/projects/f1df9f889b3642a7ac056884bc2c51a4` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:17.860535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] RBAC: Authorizing `identity:get_project(project_id=f1df9f889b3642a7ac056884bc2c51a4)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:17.862623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:17.864695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 234/452] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:08:17 2022] GET /identity/v3/projects/f1df9f889b3642a7ac056884bc2c51a4 => generated 344 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:17.871822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:17.984659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:18.033148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1165885-f331-4507-9753-8b4000e944b4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:18.081836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f1fbdc-42d0-4611-924d-3725e564e351 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:18.100146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:18.101059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:18.101497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:18.101858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:18.103035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] RBAC: Adding query filter params (group.id=1848218daa7a489394e0133a4f8d46b0, role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=64d8a5429b86463da1df922aeb084e5f) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:18.103437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:18.105349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:18.113545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 219/453] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:17 2022] GET /identity/v3/role_assignments?group.id=1848218daa7a489394e0133a4f8d46b0&scope.project.id=64d8a5429b86463da1df922aeb084e5f&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 611 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:18.143765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701ec463-88ce-427a-807b-818d0b647ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:18.144836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701ec463-88ce-427a-807b-818d0b647ff7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:18.145363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701ec463-88ce-427a-807b-818d0b647ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:18.145851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701ec463-88ce-427a-807b-818d0b647ff7 None admin] PATH_INFO: `/v3/groups/7f5d7675d60e4a0eaed89a8e62532d89` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:18.147995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701ec463-88ce-427a-807b-818d0b647ff7 None admin] RBAC: Authorizing `identity:get_group(group_id=7f5d7675d60e4a0eaed89a8e62532d89)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:18.150581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701ec463-88ce-427a-807b-818d0b647ff7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:18.153600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 235/454] 10.222.0.52 () {56 vars in 1180 bytes} [Sat Aug 27 02:08:17 2022] GET /identity/v3/groups/7f5d7675d60e4a0eaed89a8e62532d89 => generated 229 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:18.162551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:18.312271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e4e182-4f06-4c87-91c4-dcc514a68d4c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:18.364647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f0ce4d-c800-4df9-bade-4ef19c182da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:18.365301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f0ce4d-c800-4df9-bade-4ef19c182da1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:18.365584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f0ce4d-c800-4df9-bade-4ef19c182da1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:18.365825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f0ce4d-c800-4df9-bade-4ef19c182da1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:18.366577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f0ce4d-c800-4df9-bade-4ef19c182da1 None admin] RBAC: Adding query filter params (group.id=7f5d7675d60e4a0eaed89a8e62532d89, role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=f1df9f889b3642a7ac056884bc2c51a4) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:18.366870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f0ce4d-c800-4df9-bade-4ef19c182da1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:18.368324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f0ce4d-c800-4df9-bade-4ef19c182da1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:18.374866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 220/455] 10.222.0.52 () {56 vars in 1376 bytes} [Sat Aug 27 02:08:18 2022] GET /identity/v3/role_assignments?group.id=7f5d7675d60e4a0eaed89a8e62532d89&scope.project.id=f1df9f889b3642a7ac056884bc2c51a4&role.id=b83e461b8f6643fba45f427d28498c41 => generated 611 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:22.121739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb75848-7aa2-43f9-8bc4-83e83f591959 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:22.122535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb75848-7aa2-43f9-8bc4-83e83f591959 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:22.122664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb75848-7aa2-43f9-8bc4-83e83f591959 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:22.126205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 236/456] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:22.132919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6e01f-c3ce-41c1-bba8-0a1067ba78d7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:22.133689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6e01f-c3ce-41c1-bba8-0a1067ba78d7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:22.133689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6e01f-c3ce-41c1-bba8-0a1067ba78d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:22.150259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:22.157320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-13f6e01f-c3ce-41c1-bba8-0a1067ba78d7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:22.162786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-13f6e01f-c3ce-41c1-bba8-0a1067ba78d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:22.256418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 221/457] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:22.306757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a752660e-54de-4cde-97a3-592207550145 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:22.307159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a752660e-54de-4cde-97a3-592207550145 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:22.307342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a752660e-54de-4cde-97a3-592207550145 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:22.309236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 237/458] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:22.316503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d13646-fcb7-4751-9a44-35217fe55fe9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:22.316918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d13646-fcb7-4751-9a44-35217fe55fe9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:22.317374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d13646-fcb7-4751-9a44-35217fe55fe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:22.339872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c187a96-c3d5-414d-b563-76bfcdb922c9 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:22.341409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c187a96-c3d5-414d-b563-76bfcdb922c9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:22.341677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c187a96-c3d5-414d-b563-76bfcdb922c9 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:22.344585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 238/459] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:08:22.353202 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46c776d-7d97-42e2-ae24-c7c2ab6abade None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:22.353662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46c776d-7d97-42e2-ae24-c7c2ab6abade None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:22.354117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46c776d-7d97-42e2-ae24-c7c2ab6abade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:22.358288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-12d13646-fcb7-4751-9a44-35217fe55fe9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:22.359498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-12d13646-fcb7-4751-9a44-35217fe55fe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:22.388198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:22.393016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b46c776d-7d97-42e2-ae24-c7c2ab6abade None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:22.395161 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b46c776d-7d97-42e2-ae24-c7c2ab6abade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:22.518776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 239/460] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:22.565378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 222/461] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:22.860146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46c776d-7d97-42e2-ae24-c7c2ab6abade None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:22.950861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d13646-fcb7-4751-9a44-35217fe55fe9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:23.048230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46c776d-7d97-42e2-ae24-c7c2ab6abade None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.096561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:23.110513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd95024a-4c97-4cb5-b90c-8002c659a530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:23.111506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd95024a-4c97-4cb5-b90c-8002c659a530 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.111963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd95024a-4c97-4cb5-b90c-8002c659a530 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.112410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd95024a-4c97-4cb5-b90c-8002c659a530 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.115721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:23.130639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd95024a-4c97-4cb5-b90c-8002c659a530 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:23.133113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd95024a-4c97-4cb5-b90c-8002c659a530 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:23.156004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d13646-fcb7-4751-9a44-35217fe55fe9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-fd95024a-4c97-4cb5-b90c-8002c659a530 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:23.157509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:23.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:23.161608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:23.161608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:23.161608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:23.161608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 240/462] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:23.161608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684f34f-5d83-4633-96bd-84adb7a144b7 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.162617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684f34f-5d83-4633-96bd-84adb7a144b7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.162617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684f34f-5d83-4633-96bd-84adb7a144b7 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.163190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 241/463] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:23.165399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e684f34f-5d83-4633-96bd-84adb7a144b7 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:23.213848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:23.214598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.214875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.215144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.229330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:23.230853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:23.255810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:23.257365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:23.258984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:23.258984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:23.258984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:23.258984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 223/464] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:23.258984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:23.323110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e684f34f-5d83-4633-96bd-84adb7a144b7 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.374892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e240310-782c-4483-b28e-396977d82ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:23.377094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e240310-782c-4483-b28e-396977d82ff7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.379176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e240310-782c-4483-b28e-396977d82ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.379896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e240310-782c-4483-b28e-396977d82ff7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.402017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e240310-782c-4483-b28e-396977d82ff7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:23.405464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e240310-782c-4483-b28e-396977d82ff7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:23.420770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e3c86-c3fc-421f-aab5-e7f7a9a3a09f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-2e240310-782c-4483-b28e-396977d82ff7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:23.429609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:23.431534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:23.433249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:23.433249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:23.433249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:23.433249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 242/465] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:23.433249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76598dae-edf4-46bf-8082-79cb99fda06e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.433249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76598dae-edf4-46bf-8082-79cb99fda06e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.433249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76598dae-edf4-46bf-8082-79cb99fda06e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.452318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-76598dae-edf4-46bf-8082-79cb99fda06e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:23.452561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:23.453534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-76598dae-edf4-46bf-8082-79cb99fda06e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:23.479070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:23.479685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.480113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.480361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.481063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:23.481330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:23.482766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:23.498297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:23.503186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 224/466] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:23.509094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:23.551212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 243/467] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:23.552649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76598dae-edf4-46bf-8082-79cb99fda06e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:23.682937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0682e60-6f7a-4948-9284-890bd49c9b76 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.713610 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76598dae-edf4-46bf-8082-79cb99fda06e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.728151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:23.729146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.729623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.730085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.730994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:23.731495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:23.733099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:23.753287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 225/468] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:23.754575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:23.770113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:23.771269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.771551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.771979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.773044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:23.773565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:23.775847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:23.801253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 244/469] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:23.802691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:23.814943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:23.843088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:23.908810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27e13d-c8fc-427f-bc24-243be0454a9c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.952323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2919b8d-dc99-44b8-b514-89b02423034a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:23.957022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:23.957669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:23.957941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:23.958205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:23.958863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:23.960451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:23.987483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 226/470] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:08:23 2022] POST /identity/v3/users => generated 246 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:08:23.988843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:24.006836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0db038-2079-44a3-8345-56a749b8b535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:24.007856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0db038-2079-44a3-8345-56a749b8b535 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:24.008309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0db038-2079-44a3-8345-56a749b8b535 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:24.008844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0db038-2079-44a3-8345-56a749b8b535 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:24.009663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0db038-2079-44a3-8345-56a749b8b535 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:24.011259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0db038-2079-44a3-8345-56a749b8b535 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:24.057157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 245/471] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:08:23 2022] POST /identity/v3/users => generated 248 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:08:24.057157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f049b289-7494-4149-abbf-70b06e02ad71 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:24.057157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f049b289-7494-4149-abbf-70b06e02ad71 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:24.057157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f049b289-7494-4149-abbf-70b06e02ad71 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:24.057157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 246/472] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:24.057157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f049b289-7494-4149-abbf-70b06e02ad71 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:24.191517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:24.205079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04acdc1-1d7c-4250-9b89-2a6596f2a9b4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:24.238608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:24.276254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f049b289-7494-4149-abbf-70b06e02ad71 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:24.276695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a610f41-3f0b-471a-859e-bac0c7b530d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:24.277678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a610f41-3f0b-471a-859e-bac0c7b530d9 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:24.278391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a610f41-3f0b-471a-859e-bac0c7b530d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:24.278832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a610f41-3f0b-471a-859e-bac0c7b530d9 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:24.280841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a610f41-3f0b-471a-859e-bac0c7b530d9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:24.284153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a610f41-3f0b-471a-859e-bac0c7b530d9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:24.323938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 227/473] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:08:23 2022] POST /identity/v3/users => generated 248 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:08:24.326545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4737e6a5-b76c-48fb-a615-f1da4723f479 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:24.326545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4737e6a5-b76c-48fb-a615-f1da4723f479 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:24.326813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4737e6a5-b76c-48fb-a615-f1da4723f479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:24.342621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:24.343784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:24.344279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:24.344716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:24.348487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4737e6a5-b76c-48fb-a615-f1da4723f479 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:24.350172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4737e6a5-b76c-48fb-a615-f1da4723f479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:24.369842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:24.372898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:24.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:24.398030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:24.399445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:24.399445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:24.399445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:24.399798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 247/474] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:24.410728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:24.492235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 228/475] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:24.632508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a3e15-ee82-44d5-9a27-2a545059ee0a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:24.695031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:24.695835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:24.696116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:24.696396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:24.697246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:24.697733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:24.700088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:24.704164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:24.732459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 248/476] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:24.740223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4737e6a5-b76c-48fb-a615-f1da4723f479 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:24.850375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:24.877191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4737e6a5-b76c-48fb-a615-f1da4723f479 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:24.878431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:24.937634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:24.938997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:24.939466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:24.940473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:24.941950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:24.943784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:24.975984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 229/477] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:08:24 2022] POST /identity/v3/users => generated 249 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:08:25.060642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650fe518-d1b0-4195-ab30-6ea35c87d2de None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:25.118554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:25.119288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:25.119723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:25.120051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:25.120948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:25.123075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:25.138368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.sql.core [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 27 02:08:25.138368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 27 02:08:25.138368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [parameters: {'id': '794951365b064153ac1f147e128525f8', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 27 02:08:25.138368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81745) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result = self._query(query) Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application conn.query(q) Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result.read() Aug 27 02:08:25.164722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return ref.to_dict() Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 27 02:08:25.165738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application next(self.gen) Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application yield resource Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application next(self.gen) Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.session.rollback() Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.force_reraise() Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise self.value Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application session.commit() Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 27 02:08:25.166632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._prepare_impl() Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.session.flush() Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._flush(objects) Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application compat.raise_( Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application flush_context.execute() Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rec.execute(self) Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 27 02:08:25.167477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 27 02:08:25.168347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result = self._query(query) Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application conn.query(q) Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result.read() Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 27 02:08:25.170089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [parameters: {'id': '794951365b064153ac1f147e128525f8', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:25.172062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 27 02:08:25.172980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:25.174761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 249/478] 10.222.0.52 () {60 vars in 1131 bytes} [Sat Aug 27 02:08:24 2022] POST /identity/v3/roles => generated 144 bytes in 290 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:08:25.181302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:25.354618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddb92f8-489f-4cac-974c-be9cc167ec60 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:25.398912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d444e2-3bd9-4906-a047-d0023308723f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:25.400283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d444e2-3bd9-4906-a047-d0023308723f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:25.401112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d444e2-3bd9-4906-a047-d0023308723f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:25.401611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d444e2-3bd9-4906-a047-d0023308723f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:25.410234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:25.421165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d444e2-3bd9-4906-a047-d0023308723f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-36d444e2-3bd9-4906-a047-d0023308723f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:25.428380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:25.429763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:25.430999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 230/479] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:08:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:25.438530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:25.553312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:25.619658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d739eca6-9325-4ff5-806b-48811cd4bc54 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:25.667228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe0d57f-a6e4-466c-8a0d-31fdf61cfc7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:25.667228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0d57f-a6e4-466c-8a0d-31fdf61cfc7d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:25.667228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0d57f-a6e4-466c-8a0d-31fdf61cfc7d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:25.667698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0d57f-a6e4-466c-8a0d-31fdf61cfc7d None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:25.668522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe0d57f-a6e4-466c-8a0d-31fdf61cfc7d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:25.669050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe0d57f-a6e4-466c-8a0d-31fdf61cfc7d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:25.671593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe0d57f-a6e4-466c-8a0d-31fdf61cfc7d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:25.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 250/480] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:27.045166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3cd2ab-9e3d-475a-8013-18f0b1d8267c None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:27.046314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3cd2ab-9e3d-475a-8013-18f0b1d8267c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:27.046913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3cd2ab-9e3d-475a-8013-18f0b1d8267c None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:27.050160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 231/481] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:27 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:27.059398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e8921-6d67-4994-a324-6732e36140c5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:27.060836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e8921-6d67-4994-a324-6732e36140c5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:27.061668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e8921-6d67-4994-a324-6732e36140c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:27.101077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-9c9e8921-6d67-4994-a324-6732e36140c5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:27.103063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9c9e8921-6d67-4994-a324-6732e36140c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:27.217272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 251/482] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:27.449241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b93aad-2740-4a47-bfbd-830c01a04396 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:27.449952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b93aad-2740-4a47-bfbd-830c01a04396 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:27.450420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b93aad-2740-4a47-bfbd-830c01a04396 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:27.452317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 232/483] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:27.463695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b2bf1-7977-40d8-b827-8695a0ec73b2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:27.465012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b2bf1-7977-40d8-b827-8695a0ec73b2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:27.465719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b2bf1-7977-40d8-b827-8695a0ec73b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:27.506166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-256b2bf1-7977-40d8-b827-8695a0ec73b2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:27.507797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-256b2bf1-7977-40d8-b827-8695a0ec73b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:27.606901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b93aad-2740-4a47-bfbd-830c01a04396 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:27.650198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:27.743296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 252/484] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:27.902434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28b7eb-9a80-4cc0-b635-9090d0853429 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:27.903439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28b7eb-9a80-4cc0-b635-9090d0853429 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:27.904185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28b7eb-9a80-4cc0-b635-9090d0853429 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:27.919555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 253/485] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:27 2022] GET /identity => generated 270 bytes in 19 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:27.927374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d62f90-1948-4f64-95d5-7f0c2d02eb39 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:27.928817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d62f90-1948-4f64-95d5-7f0c2d02eb39 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:27.929487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d62f90-1948-4f64-95d5-7f0c2d02eb39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:27.978072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b93aad-2740-4a47-bfbd-830c01a04396 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:28.022016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-09d62f90-1948-4f64-95d5-7f0c2d02eb39 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:28.026540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-09d62f90-1948-4f64-95d5-7f0c2d02eb39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:28.107932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:28.111527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:28.112484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:28.113649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:28.145809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:28.161828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:28.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:28.163953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:28.177478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 233/486] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 563 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:28.181948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:28.233259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 254/487] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 303 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:28.233259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d62f90-1948-4f64-95d5-7f0c2d02eb39 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:28.363540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:28.390805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d62f90-1948-4f64-95d5-7f0c2d02eb39 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:28.414975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:28.460853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f31f3d-154d-4d21-a6b3-3f46f35e39bc None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:28.472588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:28.473484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:28.473887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:28.474193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:28.499616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:28.513148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:28.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:28.514990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 255/488] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:28.529160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:28.568137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:28.569631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:28.570356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:28.571066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:28.572317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:28.573133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:28.575857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:28.593849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 234/489] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 414 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:28.615009 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:28.726863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bf90b-59ab-4d4c-b955-63e246ed3402 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:28.774504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:28.775019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:28.775296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:28.775567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:28.776306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:28.776615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:28.778144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:28.798528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf2b9df-94da-4a0d-a4e8-dec5271d4fd9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:28.809264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 256/490] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:28.809264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:28.824517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:28.866766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb6136-484c-445a-9540-1c63e5047594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:28.867960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb6136-484c-445a-9540-1c63e5047594 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:28.870549 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb6136-484c-445a-9540-1c63e5047594 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:28.870549 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb6136-484c-445a-9540-1c63e5047594 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:28.873487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:28.891897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb6136-484c-445a-9540-1c63e5047594 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:28.894716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb6136-484c-445a-9540-1c63e5047594 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-c7fb6136-484c-445a-9540-1c63e5047594 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:28.918913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:28.920662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:28.922487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:28.922487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:28.922487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:28.922487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 235/491] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:28.922878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2875bd-fc58-41fb-abfa-dc2f029a5322 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:28.923573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2875bd-fc58-41fb-abfa-dc2f029a5322 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:28.923777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2875bd-fc58-41fb-abfa-dc2f029a5322 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:28.929440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 236/492] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:28.929440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2875bd-fc58-41fb-abfa-dc2f029a5322 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:29.013109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bad69d-e9a1-4a66-a3fa-3ab7374eb475 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:29.073889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:29.074554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.074923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.075198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.093155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:29.095189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2875bd-fc58-41fb-abfa-dc2f029a5322 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:29.100332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:29.101397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 257/493] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:29.106648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:29.153542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3afb16-579d-4dde-81d1-2df636c2b0d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:29.155128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3afb16-579d-4dde-81d1-2df636c2b0d2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.155826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3afb16-579d-4dde-81d1-2df636c2b0d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.156569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3afb16-579d-4dde-81d1-2df636c2b0d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.173182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3afb16-579d-4dde-81d1-2df636c2b0d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:29.175863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3afb16-579d-4dde-81d1-2df636c2b0d2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-0f3afb16-579d-4dde-81d1-2df636c2b0d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:29.198875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:29.200010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:29.201297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:29.201297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:29.201297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:29.204417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 237/494] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:29.206735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e203c-e6cb-437e-b6e2-7688cd54fcb1 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.207179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e203c-e6cb-437e-b6e2-7688cd54fcb1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.207629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e203c-e6cb-437e-b6e2-7688cd54fcb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.235811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ff1e203c-e6cb-437e-b6e2-7688cd54fcb1 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:29.237722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ff1e203c-e6cb-437e-b6e2-7688cd54fcb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:29.267283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:29.270412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:29.278985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6090389d-8fec-4ed9-a60c-03a2aad35e11 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:29.336154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:29.336830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.337275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.337659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.338403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:29.338878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:29.340811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:29.347550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 238/495] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:29.348428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1e203c-e6cb-437e-b6e2-7688cd54fcb1 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:29.356887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 258/496] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:29.357808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:29.499067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1e203c-e6cb-437e-b6e2-7688cd54fcb1 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:29.550570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636e9496-43a9-4e63-965f-536e7dc9d33f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:29.561411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128770b7-e8db-44c6-81e8-8cf8ebc67d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:29.563942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128770b7-e8db-44c6-81e8-8cf8ebc67d3c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.565062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128770b7-e8db-44c6-81e8-8cf8ebc67d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.565951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128770b7-e8db-44c6-81e8-8cf8ebc67d3c None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.567137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128770b7-e8db-44c6-81e8-8cf8ebc67d3c None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:29.567454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128770b7-e8db-44c6-81e8-8cf8ebc67d3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:29.569546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128770b7-e8db-44c6-81e8-8cf8ebc67d3c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:29.583262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 239/497] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:29.584626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49253814-f879-4368-8b79-95489181705c None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.584908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49253814-f879-4368-8b79-95489181705c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.585206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49253814-f879-4368-8b79-95489181705c None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.586990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 240/498] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:29.588060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49253814-f879-4368-8b79-95489181705c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:29.609295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:29.611485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.612520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.613339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.615021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:29.615718 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:29.616284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:29.619170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:29.644381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 259/499] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:29.646597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:29.678879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:29.766897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49253814-f879-4368-8b79-95489181705c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:29.807029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6b174b-bc8c-415b-aeba-ebc186c59b9a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:29.811919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c58295c-369e-4c6a-be10-b93f5d272308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:29.813718 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58295c-369e-4c6a-be10-b93f5d272308 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.814160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58295c-369e-4c6a-be10-b93f5d272308 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.814895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58295c-369e-4c6a-be10-b93f5d272308 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.824572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c58295c-369e-4c6a-be10-b93f5d272308 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:29.826099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c58295c-369e-4c6a-be10-b93f5d272308 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-0c58295c-369e-4c6a-be10-b93f5d272308 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:29.845636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:29.847086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:29.848158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 241/500] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:29.850311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f61f62-dc28-4f4c-ac18-6926ea362c84 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.850789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f61f62-dc28-4f4c-ac18-6926ea362c84 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.851238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f61f62-dc28-4f4c-ac18-6926ea362c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.866896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:29.867554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:29.867844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:29.868114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:29.875906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-53f61f62-dc28-4f4c-ac18-6926ea362c84 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:29.878019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-53f61f62-dc28-4f4c-ac18-6926ea362c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:29.887714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:29.890249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:29.916225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:29.917411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:29.918849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:29.918849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:29.918849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:29.918849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 260/501] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:29.919476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:29.992457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 242/502] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:29.994017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f61f62-dc28-4f4c-ac18-6926ea362c84 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.003299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:30.034841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:30.104492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1762f8-54e6-4751-bf42-b2efbf7de5de None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.176910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.177766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.178048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.178329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:30.188044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:30.190990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:30.199244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f61f62-dc28-4f4c-ac18-6926ea362c84 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:30.209474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:30.210723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 261/503] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:08:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:30.212204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.270932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.272018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.272486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.272940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:30.293214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:30.301063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:30.302775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 243/504] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:30.306920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.381187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c176f4e7-0def-4af8-b65d-340a0d9ea721 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.407732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:30.415764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:30.432877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.433884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.434301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.434709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:30.435971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:30.436434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:30.438443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:30.472349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf86a4b-e227-4668-bc16-2be7219d8933 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.492530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 262/505] 10.222.0.52 () {56 vars in 1137 bytes} [Sat Aug 27 02:08:30 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:30.493463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.525889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.526544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.526921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.527277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:30.528034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:30.528316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:30.530322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:30.552470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 244/506] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:30.553413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.648685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdd3976-23c3-4320-ab6e-5d4c71746721 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.707664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.708451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.708724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.708994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:30.709791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:30.710096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:30.711820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:30.729284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638b4ba-8440-4905-9a07-ea4bd09ff8e9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.733350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 263/507] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:08:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:30.734436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.763467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:30.777014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.777850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.778228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.778619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:30.779463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:30.779829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:30.781692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:30.789455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:30.792085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 245/508] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:30.793392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.875885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7fa109-06eb-4bc2-8bf1-7524213afb8f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.925603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3d4b5f-b5f2-4708-92b6-eb68efbb33bb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:30.927746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.929325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.929325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.929325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:30.947060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:30.948681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:30.967689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:30.970069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:30.972093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:30.972093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:30.972093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:30.972093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 264/509] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:30.972093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:30.978175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:30.978920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:30.979260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:30.979613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:31.057111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:31.077760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:31.080572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:31.151355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:31.162322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735c2bc-461c-49e7-b235-f7ba1d41b182 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:31.216880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:31.217506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:31.217777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:31.218139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:31.227935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:31.229686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:31.243865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 246/510] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:31.244926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:31.245286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:31.246835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:31.248641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 265/511] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:30 2022] GET /identity/v3/users/glance => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:31.249904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:31.372569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:31.396405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750973e7-ca64-4607-96d5-dcc132117cda None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:31.398811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d42b2-49d1-4b4e-b083-776d9f3d13c0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:31.453049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:31.453894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:31.454251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:31.454611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:31.456179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:31.457304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:31.457861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:31.458422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:31.469752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:31.471148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:31.489740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:31.491077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:31.492434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:31.492434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:31.492434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:31.492434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 266/512] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:31.492434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:31.510080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:31.583412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:31.585190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:31.659025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ecd124-2e44-4911-a2cc-4a68c0cb6fff None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:31.704452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:31.705182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:31.705469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:31.705743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:31.706516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:31.706920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:31.708477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:31.718371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:31.728349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 267/513] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:31.729669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:31.743522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 247/514] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:31.744707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:31.829844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:31.885002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7460a20d-05a9-4c3b-9150-f6a5fd0eb145 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:31.887847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c07739-3340-4a0f-8694-5f23b6c65242 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:31.938101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:31.939992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:31.939992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:31.940675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:31.942752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:31.943585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:31.944074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:31.944352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:31.946362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:31.946362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:31.947475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:31.956795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:31.959410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:31.987629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:31.989518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:31.991240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:31.991240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:31.991240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:31.991240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 268/515] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:31.991240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:31.999348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 248/516] 10.222.0.52 () {56 vars in 1137 bytes} [Sat Aug 27 02:08:31 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:32.000784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:32.070345 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:32.163059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ebcb2e-ac3f-4f2b-a665-535836392160 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:32.167472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80145b5-7163-4a5a-8e2c-616fae0c6351 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:32.199020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:32.227868 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:32.228535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:32.228815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:32.229087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:32.229894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:32.230250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:32.231734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:32.232555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:32.233488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:32.233935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:32.234366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:32.235634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:32.238725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:32.254521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 269/517] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:32.255434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:32.297165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 249/518] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:08:31 2022] POST /identity/v3/users => generated 247 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:08:32.297165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:32.429009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de53bcb-3506-4bc5-beb9-66c8b2ac97ed None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:32.477912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:32.481444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d09f9c-c887-42dd-b3d9-c625f4d3fca2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:32.502900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:32.503941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:32.504417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:32.504949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:32.529677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:32.531687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:32.556828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:32.558486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:32.561068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:32.561068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:32.561068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:32.561068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 270/519] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:32.561068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:32.561538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:32.561646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:32.561937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:32.562185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:32.563059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:32.563493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:32.565559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:32.586284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 250/520] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:32.586284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:32.627799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:32.748496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e663f1-12f6-4305-8f7f-eddea4f1bfb4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:32.798564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a70142-f73a-42a5-a3d4-5099333be47c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:32.807716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:32.808432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:32.808708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:32.809041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:32.865966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:32.866621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:32.866900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:32.867170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:32.877477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:32.879812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:32.894889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:32.896706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 251/521] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:32 2022] GET /identity/v3/users/neutron => generated 85 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:32.898564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:32.939155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:32.945433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:32.948484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:33.006371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:33.077565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595c115b-1e3c-4b41-9534-010acf9638ab None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:33.140405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 271/522] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 575 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:33.140405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:33.143468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:33.144663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:33.144663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:33.144937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:33.145736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:33.146085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:33.147933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:33.168278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 252/523] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:33.168802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:33.311647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:33.343854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3156d24-82e2-4a61-8a97-b3d42addd802 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:33.356291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7811944f-61b9-4d3d-8fd0-f120aaf33f2c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:33.397246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:33.410811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:33.412135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:33.412715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:33.413074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:33.417037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:33.417773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:33.418078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:33.418359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:33.421523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:33.421523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:33.421971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:33.423931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:33.428078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:33.453273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 253/524] 10.222.0.52 () {56 vars in 1139 bytes} [Sat Aug 27 02:08:33 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:33.453273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:33.453660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:33.455716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:33.458571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 272/525] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:33.459781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:33.640042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554815b2-8fad-47e3-8970-4e310d0061da None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:33.652437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af080396-900c-4ae7-b8ac-9a71db31120c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:33.700182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:33.700833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:33.701527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:33.701527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:33.719220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:33.721629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:33.724150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:33.726847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:33.727999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:33.728515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:33.729659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:33.743558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:33.746593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:33.750921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:33.752611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:33.763247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:33.763247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:33.763247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:33.763247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 254/526] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:33.763247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:33.768188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:33.771541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:33.775725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 273/527] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:33.778282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:33.862556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:33.921430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370db488-5705-4f89-b880-48fd9e6334d6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:33.950076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e034a7-0307-44e7-86e9-bedd8d7ef32f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:33.971604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:33.972219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:33.972497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:33.972805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:34.003755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:34.005080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:34.005563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:34.005928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:34.006943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:34.007391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:34.010023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:34.035146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 274/528] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:34.036022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:34.074941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:34.092578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:34.094739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:34.199951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b13a2f-2026-4d31-b99b-3b2a44bfc295 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:34.222695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:34.258087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:34.258703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:34.258975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:34.259242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:34.259884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:34.260180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:34.260977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 255/529] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:34.261594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:34.262559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:34.283921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 275/530] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:34.283921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:34.416015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:34.430573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296653da-7393-4297-8abf-cfc201c76dac None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:34.469607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73e487f-3ee1-4312-98c0-dee536105798 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:34.490226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:34.490933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:34.491211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:34.491487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:34.492276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:34.492547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:34.493987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:34.520560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 256/531] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:34.520560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:34.536005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:34.536787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:34.537257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:34.537686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:34.538959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=598bc689672741c0ac600e42cf45e39d) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:34.539360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:34.541894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:34.555010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 276/532] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:34 2022] GET /identity/v3/role_assignments?user.id=598bc689672741c0ac600e42cf45e39d&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 259 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:34.556352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:34.637848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:34.712690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a48d49-a748-4616-9236-8693bcbbe1c4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:34.746359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d702c69-1fb4-4fe8-b4eb-13b1f949ce2d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:34.770126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:34.770951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:34.771421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:34.771856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:34.789328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:34.791259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:34.794004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:34.794650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:34.794924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:34.795191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:34.805249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:34.806867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:34.817555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:34.818892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:34.819720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:34.819720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:34.819720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:34.819720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 257/533] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:34.820567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:34.820697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:34.822360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:34.823922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 277/534] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:34.827924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:34.830577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:34.973306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:34.982907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5758e385-67ff-48ca-aeeb-f9ea36333de1 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:34.983380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a95c1a-e07a-4647-b39d-e6935c77f16e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:35.039678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b039a7a-1b02-4028-b961-68ea0b5bb09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:35.040226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:35.040407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b039a7a-1b02-4028-b961-68ea0b5bb09e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.040676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b039a7a-1b02-4028-b961-68ea0b5bb09e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.040771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.040879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.041110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b039a7a-1b02-4028-b961-68ea0b5bb09e None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.041260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.041831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b039a7a-1b02-4028-b961-68ea0b5bb09e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:35.041960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=c83f0d0e2e8047a98fbe114ceb15c6e0) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:35.042274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b039a7a-1b02-4028-b961-68ea0b5bb09e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:35.042396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:35.043631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b039a7a-1b02-4028-b961-68ea0b5bb09e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:35.044257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:35.057519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 258/535] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:34 2022] GET /identity/v3/role_assignments?user.id=c83f0d0e2e8047a98fbe114ceb15c6e0&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 259 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:35.057519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:35.075522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 278/536] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:35.078317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4983dbee-d42d-4a24-912b-449c0e4673b4 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.078962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4983dbee-d42d-4a24-912b-449c0e4673b4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.079545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4983dbee-d42d-4a24-912b-449c0e4673b4 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.082305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 279/537] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:35.083268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4983dbee-d42d-4a24-912b-449c0e4673b4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:35.175845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:35.230266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c387395-7a39-41c8-8765-ef9d08c484ba None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:35.232245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4983dbee-d42d-4a24-912b-449c0e4673b4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:35.277537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95d0a62-d71e-4736-a1ba-1de850eeca0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:35.278602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d0a62-d71e-4736-a1ba-1de850eeca0c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.279185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d0a62-d71e-4736-a1ba-1de850eeca0c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.279706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95d0a62-d71e-4736-a1ba-1de850eeca0c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.280810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d0a62-d71e-4736-a1ba-1de850eeca0c None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=ab71b242a16641429c29dc8aa67867e5) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:35.281497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d0a62-d71e-4736-a1ba-1de850eeca0c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:35.283565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95d0a62-d71e-4736-a1ba-1de850eeca0c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:35.291377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 280/538] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:35 2022] GET /identity/v3/role_assignments?user.id=ab71b242a16641429c29dc8aa67867e5&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 259 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:35.293113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64718c9d-0f66-47a7-bfdb-dbb590fdbb99 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.293654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64718c9d-0f66-47a7-bfdb-dbb590fdbb99 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.294299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64718c9d-0f66-47a7-bfdb-dbb590fdbb99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.298785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:35.300253 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.300685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.301110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.302209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:35.302683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:35.305052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:35.324527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-64718c9d-0f66-47a7-bfdb-dbb590fdbb99 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:35.326511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-64718c9d-0f66-47a7-bfdb-dbb590fdbb99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:35.328848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 259/539] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:35.339297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:35.369694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:35.500673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 281/540] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:35.584216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797ae7c3-5b21-4017-aca2-9ca313eaa66b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:35.623003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:35.635710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:35.636605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.636879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.637161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.648385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:35.651272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:35.680479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:35.682019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:35.683413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 260/541] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:35.695451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64718c9d-0f66-47a7-bfdb-dbb590fdbb99 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:35.917041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64718c9d-0f66-47a7-bfdb-dbb590fdbb99 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:35.974228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547fc871-38c1-4820-a606-ee744600745d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:35.974859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547fc871-38c1-4820-a606-ee744600745d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:35.975135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547fc871-38c1-4820-a606-ee744600745d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:35.975406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547fc871-38c1-4820-a606-ee744600745d None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:35.976127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547fc871-38c1-4820-a606-ee744600745d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:35.976428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547fc871-38c1-4820-a606-ee744600745d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:35.978024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547fc871-38c1-4820-a606-ee744600745d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:36.004576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 282/542] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:36.012510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:36.134456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547fc871-38c1-4820-a606-ee744600745d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:36.161520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:36.338283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527db35f-ffae-4e27-ad5d-16a9f21133e0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:36.339356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547fc871-38c1-4820-a606-ee744600745d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:36.405053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:36.405733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:36.406001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:36.406303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:36.435154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:36.455376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:36.456301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:36.456730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:36.457169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:36.458382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=56aaa029a9c542e0bc4658c5cd55c2b9) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:36.458865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:36.460177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:36.461019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:36.461832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:36.462948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 283/543] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:08:36.464118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:36.470051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:36.492303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 261/544] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:36 2022] GET /identity/v3/role_assignments?user.id=56aaa029a9c542e0bc4658c5cd55c2b9&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 259 bytes in 481 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:36.582350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:36.631091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb22a5b5-2ad7-4dea-bd51-d56e4a0c6759 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:36.685804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:36.686532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:36.686863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:36.687133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:36.688403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:36.689073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:36.692506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:36.708129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 284/545] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:08:36.715451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:36.882617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:37.011962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ba0162-c728-4eed-a3b2-f0fdf7fc056c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:37.075673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:37.076343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:37.076620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:37.076894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:37.097673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:37.099496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:37.123306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:37.125737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:37.127791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:37.127791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:37.127791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:37.127791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 262/546] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 410 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:37.139039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:37.373361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d0c989-cc5f-4899-8a1c-06f2aa6123c5 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:37.435788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:37.450344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9888b162-e23a-42f0-91a5-4a51c9533646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:37.451370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9888b162-e23a-42f0-91a5-4a51c9533646 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:37.451823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9888b162-e23a-42f0-91a5-4a51c9533646 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:37.452259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9888b162-e23a-42f0-91a5-4a51c9533646 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:37.454013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9888b162-e23a-42f0-91a5-4a51c9533646 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:37.454013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9888b162-e23a-42f0-91a5-4a51c9533646 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:37.455461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9888b162-e23a-42f0-91a5-4a51c9533646 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:37.479111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 285/547] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:37.485970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:37.635895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04b592f-8af8-4940-94db-1e36b3654b8b None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:37.639454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04b592f-8af8-4940-94db-1e36b3654b8b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:37.639454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04b592f-8af8-4940-94db-1e36b3654b8b None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:37.639454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 286/548] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:08:37.650767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac80474-c2d1-4479-b717-00d3e701bd04 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:37.651439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac80474-c2d1-4479-b717-00d3e701bd04 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:37.652032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac80474-c2d1-4479-b717-00d3e701bd04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:37.665707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd96f05a-91d3-4639-a0e3-35339ad6b837 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:37.669455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:37.697325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5ac80474-c2d1-4479-b717-00d3e701bd04 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:37.698473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5ac80474-c2d1-4479-b717-00d3e701bd04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:37.732397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85f7d50-6ed7-47ae-8c91-9e2ad923856d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:37.733065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f7d50-6ed7-47ae-8c91-9e2ad923856d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:37.733342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f7d50-6ed7-47ae-8c91-9e2ad923856d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:37.733614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f7d50-6ed7-47ae-8c91-9e2ad923856d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:37.748459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85f7d50-6ed7-47ae-8c91-9e2ad923856d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:37.749988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85f7d50-6ed7-47ae-8c91-9e2ad923856d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f85f7d50-6ed7-47ae-8c91-9e2ad923856d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:37.773951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:37.774929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:08:37.775830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:37.778794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 263/549] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:08:37 2022] GET /identity/v3/users/swift => generated 83 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:37.780522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f9209-ea6a-4063-99b5-83fc726ad331 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:37.781456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f9209-ea6a-4063-99b5-83fc726ad331 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:37.781998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f9209-ea6a-4063-99b5-83fc726ad331 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:37.787872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 264/550] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:37.789000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930f9209-ea6a-4063-99b5-83fc726ad331 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:37.833203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 287/551] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:37.833203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ac45d-4413-4f99-9ab5-84e973e1961f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:37.833203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ac45d-4413-4f99-9ab5-84e973e1961f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:37.833203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ac45d-4413-4f99-9ab5-84e973e1961f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:37.856309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-de5ac45d-4413-4f99-9ab5-84e973e1961f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:37.858293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-de5ac45d-4413-4f99-9ab5-84e973e1961f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:37.993334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 288/552] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:38.072416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930f9209-ea6a-4063-99b5-83fc726ad331 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:38.150323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:38.151641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:38.152295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:38.152941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:38.154643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:38.155402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:38.158458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:38.202772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 265/553] 10.222.0.52 () {56 vars in 1135 bytes} [Sat Aug 27 02:08:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 414 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:38.211322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5ac45d-4413-4f99-9ab5-84e973e1961f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:38.331736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:38.351324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:38.384570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:38.493813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5ac45d-4413-4f99-9ab5-84e973e1961f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:38.567222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:38.567968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8a086-3cb7-4255-95d4-984ea3c623ab None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:38.568702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:38.569396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:38.570020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:38.626185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934f583e-79ca-4294-a94b-46019678ffae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:38.627084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f583e-79ca-4294-a94b-46019678ffae None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:38.627520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f583e-79ca-4294-a94b-46019678ffae None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:38.627934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f583e-79ca-4294-a94b-46019678ffae None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:38.642085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934f583e-79ca-4294-a94b-46019678ffae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-934f583e-79ca-4294-a94b-46019678ffae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:38.648403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:38.650316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 266/554] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:08:38.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a5dff9-e700-4dd6-b129-5cca35e6e842 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:38.660726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a5dff9-e700-4dd6-b129-5cca35e6e842 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:38.660726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a5dff9-e700-4dd6-b129-5cca35e6e842 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:38.660726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 267/555] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:38.660726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a5dff9-e700-4dd6-b129-5cca35e6e842 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:38.701454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:38.703375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:38.782602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:38.796169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:38.831059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a5dff9-e700-4dd6-b129-5cca35e6e842 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:38.870419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 289/556] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 660 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 27 02:08:38.871823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:38.886027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93188977-ac5b-4d53-98b2-366c0fc8fb4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:38.886586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93188977-ac5b-4d53-98b2-366c0fc8fb4a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:38.886973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93188977-ac5b-4d53-98b2-366c0fc8fb4a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:38.887264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93188977-ac5b-4d53-98b2-366c0fc8fb4a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:38.903260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93188977-ac5b-4d53-98b2-366c0fc8fb4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-93188977-ac5b-4d53-98b2-366c0fc8fb4a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:38.910677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:38.912511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 268/557] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:38.914060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdba6d7-191b-487b-8199-8d0b6ad8ab94 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:38.915497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdba6d7-191b-487b-8199-8d0b6ad8ab94 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:38.915497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdba6d7-191b-487b-8199-8d0b6ad8ab94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:38.940771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4cdba6d7-191b-487b-8199-8d0b6ad8ab94 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:38.942807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4cdba6d7-191b-487b-8199-8d0b6ad8ab94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:39.053961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8afc206-05cd-4b54-b7d8-12308e2e9f73 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:39.057607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 269/558] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:39.057607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdba6d7-191b-487b-8199-8d0b6ad8ab94 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:39.120683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:39.121748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.122249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.122733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.124076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:39.124659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:39.128088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:39.142256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 290/559] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:39.143587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:39.172269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:39.182338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:39.240934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdba6d7-191b-487b-8199-8d0b6ad8ab94 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:39.313164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:39.314957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.315731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.316423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.335561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:39.339620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:39.345299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db60e8-34c0-4cd4-82d5-99baf60d0e2f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:39.368969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:39.373760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:39.375629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:39.375629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:39.375629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:39.375629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 270/560] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:39.375629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:39.413701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:39.414343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.414621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.414888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.415573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:39.415869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:39.417271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:39.426762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 291/561] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:39.433042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:39.537018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f40aa90-ced8-4bee-99e1-595f664effc4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:39.553555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:39.577988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:39.595745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:39.596558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.596965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.597400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.597761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3c64f-e270-4c1d-b5c5-e70efad8229b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:39.620652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:39.622154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:39.649110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:39.650968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:39.652579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:39.652579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:39.652579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:39.652579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 271/562] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:39.652579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:39.660017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:39.661009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.661495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.661943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.678855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:39.684541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:39.685647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:39.686454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 292/563] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:39.687123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:39.809745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d218d75c-5f02-4456-8983-220021eefdb1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:39.861654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27525e0a-32f1-4416-bf97-e76741e3e112 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:39.871471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b441d1de-3fc0-4056-bf4c-d6251de77294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:39.872095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441d1de-3fc0-4056-bf4c-d6251de77294 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.872367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441d1de-3fc0-4056-bf4c-d6251de77294 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.872635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441d1de-3fc0-4056-bf4c-d6251de77294 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.873341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b441d1de-3fc0-4056-bf4c-d6251de77294 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:39.873641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b441d1de-3fc0-4056-bf4c-d6251de77294 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:39.875139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b441d1de-3fc0-4056-bf4c-d6251de77294 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:39.890194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 272/564] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:39.891674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803afb8-ca47-44f8-91a3-d7afbec4158f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.891982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803afb8-ca47-44f8-91a3-d7afbec4158f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.892327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803afb8-ca47-44f8-91a3-d7afbec4158f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.894137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 273/565] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:39.894886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1803afb8-ca47-44f8-91a3-d7afbec4158f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:39.933269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:39.934196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:39.934546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:39.935043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:39.938249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:39.947953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:39.955341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:39.958235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:39.978667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:39.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:39.982393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:39.982393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:39.982393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:39.982393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 293/566] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:39.982393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:40.067282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1803afb8-ca47-44f8-91a3-d7afbec4158f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:40.130125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:40.131050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.131613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:40.131916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:40.133046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:40.133646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:40.135982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:40.151223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 274/567] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:40.151223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:40.168919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d8ef6-72d0-4b3b-9d10-52f1da7e8cb1 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:40.232016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529cf80f-afbb-4a35-8e79-7d1208f4ba62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:40.232016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529cf80f-afbb-4a35-8e79-7d1208f4ba62 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.232016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529cf80f-afbb-4a35-8e79-7d1208f4ba62 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:40.232016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529cf80f-afbb-4a35-8e79-7d1208f4ba62 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:40.233044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529cf80f-afbb-4a35-8e79-7d1208f4ba62 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:40.233044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529cf80f-afbb-4a35-8e79-7d1208f4ba62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:40.233788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529cf80f-afbb-4a35-8e79-7d1208f4ba62 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:40.242850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 294/568] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:40.244115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdc53f4-eee1-4c15-8d1a-b303848c4174 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.244392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdc53f4-eee1-4c15-8d1a-b303848c4174 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:40.245062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdc53f4-eee1-4c15-8d1a-b303848c4174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:40.267275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4fdc53f4-eee1-4c15-8d1a-b303848c4174 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:40.269055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4fdc53f4-eee1-4c15-8d1a-b303848c4174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:40.330239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:40.339508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929beb36-9480-47e1-87a3-ec45ee2e3f16 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:40.350642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:40.384786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 295/569] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:40.385958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdc53f4-eee1-4c15-8d1a-b303848c4174 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:40.396807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:40.397891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.398343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:40.398803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:40.416799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:40.419954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:40.446696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:40.459834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 275/570] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:40.463414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:40.594027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdc53f4-eee1-4c15-8d1a-b303848c4174 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:40.651635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:40.652494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.652949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:40.653462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:40.654518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:40.654857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:40.656417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:40.664845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8075744-442f-4d44-8527-f1d3a335d5e0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:40.684323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 296/571] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:40.684323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:40.735073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:40.736074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.736672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:40.737278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:40.750018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:40.752366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:40.752644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:40.773590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:40.775300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 276/572] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:40 2022] GET /identity/v3/users/cinder => generated 84 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:40.778341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:40.808058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:40.873627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219fe77-8880-41dc-a4b0-b0f8413ad1ab None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:40.929990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:40.930935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.931361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:40.931783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:40.935566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c4bbb2-398b-4521-a164-d9a9dab952a3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:40.950504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:40.952747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:40.974596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:40.976489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:40.978295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:40.978295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:40.978295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:40.978295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 297/573] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:40.978295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:40.998747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:40.999489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:40.999937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.000391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.001411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:41.001801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.003544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:41.021782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 277/574] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:41.022654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.119598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:41.138638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b89623-2e88-4aaa-95e3-19344e11e205 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.155617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:41.197643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.198314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.198787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.199265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.201815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4a92b0-ea85-4d2c-b408-12071788b460 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.213413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.215649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:41.232220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:41.234128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 298/575] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:08:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:41.235847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.261405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.262022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.262297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.262612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.263382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:41.263685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.265184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:41.289465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 278/576] 10.222.0.52 () {56 vars in 1137 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:41.290423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.390700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d453083b-f61f-4504-bccb-80679df3b85b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.433891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4baca6f-b037-4616-b7f1-5d9b2ef8933a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.438546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.439215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.439782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.440047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.458000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:41.470514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:41.471709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 299/577] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:41.473368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.476342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:41.486963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.487700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.488117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.488501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.489764 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:41.490094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.491590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:41.492884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:41.506399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 279/578] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:41.507258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.621902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b4e7b-528d-43c5-b3c7-b3e2287ade88 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.644417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995ca1ac-2d2b-4739-a3d8-822c80282b60 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.672524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.673252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.673572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.673904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.674712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=cfc9a334f972474aa19aa3beae51560f) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:41.675128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.676766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:41.684413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 300/579] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/role_assignments?user.id=cfc9a334f972474aa19aa3beae51560f&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 259 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:41.686472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.696577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.697287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.697552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.697788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.698570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:41.698879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.700383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:41.740262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 280/580] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:41.740262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.829206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:41.829839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:41.872190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cffdf5-144f-427a-8bc7-3ef7b86f6994 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.918790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.919800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.919800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.920282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.932984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a104b55-d073-44a0-a841-fb6311fd09ad None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:41.937784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.939547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:41.969434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:41.970532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:41.971685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:41.971685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:41.971685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:41.978438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 301/581] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:41.979862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:41.991027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:41.992022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:41.992290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:41.992554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:41.993364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:41.993671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:41.995092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:42.012014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 281/582] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:42.012996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:42.151068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04de2d33-1f07-4ac3-8714-b873719ed54e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.161388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7544e8b9-b8ce-402c-bef9-8c3119260d85 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.177584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:42.195586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:42.206596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:42.207414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:42.207870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:42.208273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:42.211487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:42.212482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:42.212922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:42.213417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] PATH_INFO: `/v3/users/glance` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:42.222440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:42.223920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:42.226614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:42.229062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:42.240821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:42.244262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:42.246004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:42.246004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:42.246004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:42.246004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 282/583] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:42.246004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:42.253256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:42.263607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 302/584] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:41 2022] GET /identity/v3/users/glance => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:42.265435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:42.427941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c14aa3-a685-4e71-b9d1-6800b809570f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.429045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b99b0-c877-4e18-b3b0-da75172ec7ab None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.486578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:42.487492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:42.487940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:42.488390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:42.489512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:42.489984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:42.490102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:42.490971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:42.491383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:42.491807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:42.492319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:42.509967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:42.512313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:42.518978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 283/585] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:42.518978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:42.542538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:42.544228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:42.547082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:42.547082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:42.547082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:42.547082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 303/586] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:42.547082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:42.555809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:42.637106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:42.680078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddba0a22-b405-4295-8deb-11af977113b2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.733502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:42.734286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:42.734589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:42.734847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:42.735598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:42.735928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:42.738140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:42.743559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8ae28-0193-4af6-9227-8b6a446af162 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.754109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 284/587] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:42.755208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:42.804947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:42.805364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:42.805786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:42.806252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:42.807650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:42.808165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:42.810401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:42.835216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 304/588] 10.222.0.52 () {56 vars in 1137 bytes} [Sat Aug 27 02:08:42 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:42.836090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:42.900672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:42.927758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d74e3-c3d1-4428-82fd-91da196f0fe0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.972205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:42.979963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700aadc2-e953-4760-b0b3-17533360ccc9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:42.992087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:42.993078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:42.993502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:42.993702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.005602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.007082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:43.021873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:43.023106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 285/589] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:43.024393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.029940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.030607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.030880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.031284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.032479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:43.032783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.034208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.049166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 305/590] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:43.050036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.191621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be243b75-a76f-4f62-a756-f913618fec11 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.196049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82b026c-8611-4b21-8a8e-7fdfc587d02e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.237154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.237449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.237988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.238161 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.238414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.238567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.238724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.239000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.247438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.249442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.252651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.254545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:43.263901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:43.264898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:43.265897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:43.270082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 286/591] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:43.271030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:43.280620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:43.284334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:43.289053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:43.289053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:43.289053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:43.289053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 306/592] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:43.289053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.292683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:43.300223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:43.444646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8951952-5ef6-4e68-9751-a5a0cc579f07 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.467660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e1f438-5aac-4149-a4aa-d0e78f37bb44 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.504006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.504573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.504842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.505102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.515473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.516927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.523016 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.524227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.524804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.525487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.527043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:43.527736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.530354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:43.531394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:43.532953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 307/593] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:43 2022] GET /identity/v3/users/neutron => generated 85 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:43.534577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.553593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 287/594] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:43.554455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.642372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:43.668715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:43.695914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6d43d1-7249-4fec-87a2-a3c25da9a4e4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.734712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c5e956-61f2-47c7-9cc3-ae492b76c2fa None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.756011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.756535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.756872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.757250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.759212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:43.759212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.762067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.783053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 308/595] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:43.784739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.793638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.794433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.794846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.795927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.796790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:43.797916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:43.799446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:43.825496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 288/596] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:43.826488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:43.940947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e25756b-ab8e-45de-9e54-fd555360fd6f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.975723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b259cb-646f-466b-a6a1-eb18a88424e0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:43.986578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:43.997394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:43.998126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:43.998389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:43.998654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:43.999392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:43.999730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:44.002064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:44.008613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:44.027006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 309/597] 10.222.0.52 () {56 vars in 1139 bytes} [Sat Aug 27 02:08:43 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:44.027006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:44.032529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:44.034235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:44.034667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:44.035137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/598bc689672741c0ac600e42cf45e39d/roles/439d280788384429bc6d5ff11a303b76` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:44.057822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=598bc689672741c0ac600e42cf45e39d, role_id=439d280788384429bc6d5ff11a303b76)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:44.059309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:44.093314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 289/598] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:08:43 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/598bc689672741c0ac600e42cf45e39d/roles/439d280788384429bc6d5ff11a303b76 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:44.094945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:44.195971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cad0bc7-0cd3-45b1-b39c-e1b8e7a2a5e8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:44.243315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:44.244012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:44.244428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:44.244871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/c83f0d0e2e8047a98fbe114ceb15c6e0/roles/439d280788384429bc6d5ff11a303b76` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:44.268629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=c83f0d0e2e8047a98fbe114ceb15c6e0, role_id=439d280788384429bc6d5ff11a303b76)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:44.270088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:44.282895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe1034-976a-46e0-8725-7286cc30b100 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:44.293157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 310/599] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:08:44 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/c83f0d0e2e8047a98fbe114ceb15c6e0/roles/439d280788384429bc6d5ff11a303b76 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:44.293157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:44.349869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:44.350908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:44.351360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:44.351804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:44.363863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:44.365848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:44.384227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:44.386115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:44.387722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 290/600] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:44.392503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:44.399276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:44.474073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:44.585209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339c01f-e60c-4b67-90ed-b64c02d6c5e0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:44.600062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb11426-d648-4d08-b5bd-ff72abf9e589 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:44.654736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:44.655724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:44.656285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:44.656707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:44.676493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:44.677089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c901387-2795-4cb1-b12a-40f3d7ae9710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:44.678275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c901387-2795-4cb1-b12a-40f3d7ae9710 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:44.678919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c901387-2795-4cb1-b12a-40f3d7ae9710 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:44.679168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:44.679727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c901387-2795-4cb1-b12a-40f3d7ae9710 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:44.681224 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c901387-2795-4cb1-b12a-40f3d7ae9710 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:44.681953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c901387-2795-4cb1-b12a-40f3d7ae9710 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:44.684461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c901387-2795-4cb1-b12a-40f3d7ae9710 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:44.707458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 291/601] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:44.713777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:44.715622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:44.717386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:44.717386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:44.717386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:44.717386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 311/602] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 427 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:44.718180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:44.719268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b008ecb-93b5-44b1-baaa-53045c20d20f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:44.719752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b008ecb-93b5-44b1-baaa-53045c20d20f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:44.720233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b008ecb-93b5-44b1-baaa-53045c20d20f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:44.722871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 292/603] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:44 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:44.726708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b008ecb-93b5-44b1-baaa-53045c20d20f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:44.866804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:44.884785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ac493e-a994-4edd-a53a-bf77afe5a2e8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:44.924279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:44.939866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a83f716-c57a-468d-9eed-b78ac89a6b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:44.940926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a83f716-c57a-468d-9eed-b78ac89a6b05 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:44.941436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a83f716-c57a-468d-9eed-b78ac89a6b05 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:44.941888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a83f716-c57a-468d-9eed-b78ac89a6b05 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/ab71b242a16641429c29dc8aa67867e5/roles/439d280788384429bc6d5ff11a303b76` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:44.942596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b008ecb-93b5-44b1-baaa-53045c20d20f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:44.970598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a83f716-c57a-468d-9eed-b78ac89a6b05 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=ab71b242a16641429c29dc8aa67867e5, role_id=439d280788384429bc6d5ff11a303b76)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:44.972562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a83f716-c57a-468d-9eed-b78ac89a6b05 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:45.005239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 312/604] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:08:44 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/ab71b242a16641429c29dc8aa67867e5/roles/439d280788384429bc6d5ff11a303b76 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:45.007658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ee17a-43b6-4c05-a721-673be0d2f021 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:45.008231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ee17a-43b6-4c05-a721-673be0d2f021 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:45.008776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ee17a-43b6-4c05-a721-673be0d2f021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:45.023108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:45.027023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:45.027857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:45.029339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:45.032092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:45.034693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:45.041635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:45.078134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-614ee17a-43b6-4c05-a721-673be0d2f021 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:45.080410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-614ee17a-43b6-4c05-a721-673be0d2f021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:45.086491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 293/605] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:45.093159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:45.199388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 313/606] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:45.323990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c18ad5-7633-4cce-9009-0051f5169feb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:45.387245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:45.387913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:45.388193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:45.388621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:45.401010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:45.403314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:45.424775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:45.427202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:45.429283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 294/607] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:08:45.435164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ee17a-43b6-4c05-a721-673be0d2f021 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:45.583404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:45.647659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ee17a-43b6-4c05-a721-673be0d2f021 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:45.709002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:45.710256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:45.710784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:45.711507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:45.712649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:45.713413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:45.716109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:45.720997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:45.736191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 314/608] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:45.754064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:45.774642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:46.019506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fb63e9-ae26-4134-9461-711d212ddbb2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:46.059478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eaa087-c2bc-4018-8ba7-de5bf129928f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:46.084290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:46.085088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:46.085421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:46.085696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:46.111842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:46.122641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:46.124536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:46.132022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 295/609] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 405 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:46.135253 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:46.141618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:46.167105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:46.168132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:46.168581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:46.169013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/56aaa029a9c542e0bc4658c5cd55c2b9/roles/439d280788384429bc6d5ff11a303b76` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:46.215269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=56aaa029a9c542e0bc4658c5cd55c2b9, role_id=439d280788384429bc6d5ff11a303b76)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:46.219441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:46.256635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 315/610] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:08:45 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/56aaa029a9c542e0bc4658c5cd55c2b9/roles/439d280788384429bc6d5ff11a303b76 => generated 0 bytes in 507 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:46.336529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:46.405025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9054d610-8a90-42ea-8ccf-12ef47bf0e70 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:46.482145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:46.483099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:46.483523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:46.484047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:46.485265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:46.485696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:46.488073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:46.508153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 296/611] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 376 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:46.517163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:46.725683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245164fd-a5c2-431f-8fd2-507a81283a4e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:46.789845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:46.790731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:46.791093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:46.791371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:46.810219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:46.812722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:46.850373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:46.852183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:46.855095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:46.855095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:46.855095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:46.855095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 316/612] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:46.862335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:47.025500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e112bd97-ecc2-43e5-9705-ee0c35ed29b7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:47.078591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:47.079353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:47.079630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:47.079904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:47.080601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:47.080956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:47.082865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:47.086747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:47.110083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 297/613] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:47.113483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:47.222796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:47.351727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1c1e98-2ef9-431b-aecd-57b6162296e9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:47.408526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb4cbd-042c-443e-869a-a47ecb6e8a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:47.409478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb4cbd-042c-443e-869a-a47ecb6e8a76 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:47.409814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb4cbd-042c-443e-869a-a47ecb6e8a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:47.410165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb4cbd-042c-443e-869a-a47ecb6e8a76 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:47.422637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb4cbd-042c-443e-869a-a47ecb6e8a76 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:47.423637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb4cbd-042c-443e-869a-a47ecb6e8a76 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-36bb4cbd-042c-443e-869a-a47ecb6e8a76 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:47.440661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:47.443801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:47.445979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 317/614] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:08:47 2022] GET /identity/v3/users/swift => generated 83 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:47.454940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:47.566192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dca99-6de9-472f-b768-896b9050186a None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:47.566821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dca99-6de9-472f-b768-896b9050186a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:47.567402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dca99-6de9-472f-b768-896b9050186a None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:47.571745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 318/615] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:47.574071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57010597-bc1c-4dbf-bfbd-56fca8de6674 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:47.574563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57010597-bc1c-4dbf-bfbd-56fca8de6674 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:47.575177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57010597-bc1c-4dbf-bfbd-56fca8de6674 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:47.577741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 319/616] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:47.587333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e57c7b6-f923-42d0-b302-9f75ea3e497a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:47.587685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e57c7b6-f923-42d0-b302-9f75ea3e497a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:47.588059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e57c7b6-f923-42d0-b302-9f75ea3e497a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:47.619626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2e57c7b6-f923-42d0-b302-9f75ea3e497a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:47.621633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2e57c7b6-f923-42d0-b302-9f75ea3e497a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:47.697719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a09b5f-013a-48c4-8510-fea376d908d5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:47.764531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 320/617] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:47.764531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ecd528-61e4-4cf5-8ffb-bedc76467080 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:47.764531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ecd528-61e4-4cf5-8ffb-bedc76467080 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:47.765044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ecd528-61e4-4cf5-8ffb-bedc76467080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:47.790687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:47.792584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:47.792855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:47.793152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:47.793983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:47.794290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:47.801042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:47.811299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b3ecd528-61e4-4cf5-8ffb-bedc76467080 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:47.820069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b3ecd528-61e4-4cf5-8ffb-bedc76467080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:47.857405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 298/618] 10.222.0.52 () {56 vars in 1135 bytes} [Sat Aug 27 02:08:47 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 403 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:47.865500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:47.866856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:47.966835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:48.026498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c5adda-6da3-423d-9229-210127fc01bf None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:48.036638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 321/619] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:48.127817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:48.128715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:48.129140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:48.129562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:48.155760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:48.158146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:48.197460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:48.198576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:48.199455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:48.199455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:48.199455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:48.200055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 299/620] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:48.212789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ecd528-61e4-4cf5-8ffb-bedc76467080 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:48.277431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:48.516244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ecd528-61e4-4cf5-8ffb-bedc76467080 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:48.549045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04439fc-1aaa-4ef2-9ae1-7e7a00b122c4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:48.559871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:48.613012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:48.614283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:48.614887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:48.615482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:48.617688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:48.618829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:48.619499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:48.620112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:48.621029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:48.621758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:48.623719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:48.631297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:48.636733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:48.638240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:48.639772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:48.643665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 300/621] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 369 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:48.644649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:48.659771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 322/622] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 444 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:48.661765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:48.686379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:48.826166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c02b1-c207-441e-9bf2-4aac112237cd None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:48.861323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2e1fa7-07e5-47ce-9b56-f85e1f334f62 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:48.893799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:48.894803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:48.895146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:48.895548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:48.913601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:48.923500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:48.927854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 301/623] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:48.928709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:48.934420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eba05c-b06d-482f-8fd4-63da5dae4a15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:48.938400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eba05c-b06d-482f-8fd4-63da5dae4a15 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:48.938400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eba05c-b06d-482f-8fd4-63da5dae4a15 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:48.938400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eba05c-b06d-482f-8fd4-63da5dae4a15 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:48.939243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eba05c-b06d-482f-8fd4-63da5dae4a15 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:48.939952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eba05c-b06d-482f-8fd4-63da5dae4a15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:48.941603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eba05c-b06d-482f-8fd4-63da5dae4a15 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:48.959987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 323/624] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:48.959987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f137d1-57d8-4271-9308-c06eca4527a0 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:48.959987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f137d1-57d8-4271-9308-c06eca4527a0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:48.959987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f137d1-57d8-4271-9308-c06eca4527a0 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:48.961095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 324/625] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:48 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:48.963464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f137d1-57d8-4271-9308-c06eca4527a0 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:48.994437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:49.119274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:49.123814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4b48dc-96da-4b2c-b5e4-b84c0dfc206e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:49.182012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f137d1-57d8-4271-9308-c06eca4527a0 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:49.184480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:49.185273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.185674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.186073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.195251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:49.196921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:49.216706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:49.218552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:49.220643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:49.225709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 302/626] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:49.227008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:49.253712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34596f3c-c95f-4068-bd2c-920e9919972b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:49.254753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34596f3c-c95f-4068-bd2c-920e9919972b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.255178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34596f3c-c95f-4068-bd2c-920e9919972b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.255632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34596f3c-c95f-4068-bd2c-920e9919972b None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.256840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34596f3c-c95f-4068-bd2c-920e9919972b None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:49.257394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34596f3c-c95f-4068-bd2c-920e9919972b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:49.259992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34596f3c-c95f-4068-bd2c-920e9919972b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:49.281577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 325/627] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:49.281577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18323c0-ac60-4e11-aa33-a46f2f610d7c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.281577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18323c0-ac60-4e11-aa33-a46f2f610d7c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.281577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18323c0-ac60-4e11-aa33-a46f2f610d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.314076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d18323c0-ac60-4e11-aa33-a46f2f610d7c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:49.316500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d18323c0-ac60-4e11-aa33-a46f2f610d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:49.364715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:49.387426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1b3277-fa53-4f6e-89f0-7024a5328b30 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:49.444568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:49.445473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.445930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.446314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.469153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 326/628] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:49.469153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18323c0-ac60-4e11-aa33-a46f2f610d7c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:49.471746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:49.473541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:49.494397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:49.495542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:49.496458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:49.496458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:49.496458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:49.497178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 303/629] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:49.498251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:49.587267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:49.655890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b6d37-e08d-4fb4-bddc-72417108e178 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:49.662657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18323c0-ac60-4e11-aa33-a46f2f610d7c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:49.715132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69ff4c-ffb0-4f2d-8304-29b8316042fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:49.716216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69ff4c-ffb0-4f2d-8304-29b8316042fb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.717152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69ff4c-ffb0-4f2d-8304-29b8316042fb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.717152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69ff4c-ffb0-4f2d-8304-29b8316042fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.717416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:49.718214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.718707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.718707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.720664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:49.721175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:49.724166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:49.730437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:49.738237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69ff4c-ffb0-4f2d-8304-29b8316042fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:49.741320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69ff4c-ffb0-4f2d-8304-29b8316042fb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:49.746544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 327/630] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:49.748017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-aa69ff4c-ffb0-4f2d-8304-29b8316042fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:49.770595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:49.772454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:49.774228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:49.774228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:49.774228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:49.779170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 304/631] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:49.780980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e1e9c-f8c0-46f7-8732-277ee3412e84 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.786032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e1e9c-f8c0-46f7-8732-277ee3412e84 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.786032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e1e9c-f8c0-46f7-8732-277ee3412e84 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.786032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 305/632] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:49.786032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e1e9c-f8c0-46f7-8732-277ee3412e84 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:49.898518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8565d2-9293-442d-ae1e-ef1c0dd12308 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:49.930290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:49.947486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:49.948464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:49.948920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:49.949397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:49.949559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e1e9c-f8c0-46f7-8732-277ee3412e84 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:49.950523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:49.951020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:49.953766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:49.974864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 328/633] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:49.975759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:50.001291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:50.002220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:50.002626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:50.003123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/cfc9a334f972474aa19aa3beae51560f/roles/439d280788384429bc6d5ff11a303b76` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:50.026035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=cfc9a334f972474aa19aa3beae51560f, role_id=439d280788384429bc6d5ff11a303b76)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:50.028032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:50.044870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 306/634] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:08:49 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/cfc9a334f972474aa19aa3beae51560f/roles/439d280788384429bc6d5ff11a303b76 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:08:50.045796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:50.116753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:50.153105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f4e56-4958-4ebe-985b-5ec103a5aa61 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:50.222791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea1f7c-4454-46ac-acc9-bdbac928d8b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:50.223717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea1f7c-4454-46ac-acc9-bdbac928d8b9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:50.224182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea1f7c-4454-46ac-acc9-bdbac928d8b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:50.224664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea1f7c-4454-46ac-acc9-bdbac928d8b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:50.225470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd78a36-44a1-4916-85f7-d6983506e2a6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:50.225942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea1f7c-4454-46ac-acc9-bdbac928d8b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:50.226328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea1f7c-4454-46ac-acc9-bdbac928d8b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:50.228774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea1f7c-4454-46ac-acc9-bdbac928d8b9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:50.247869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 329/635] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:50.250874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21839f31-cfb2-40e9-9313-144aa0bfa527 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:50.251484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21839f31-cfb2-40e9-9313-144aa0bfa527 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:50.251991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21839f31-cfb2-40e9-9313-144aa0bfa527 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:50.282405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-21839f31-cfb2-40e9-9313-144aa0bfa527 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:50.284298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-21839f31-cfb2-40e9-9313-144aa0bfa527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:50.289330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:50.290323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:50.290793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:50.291245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:50.315704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:50.321897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:50.323174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 307/636] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:50.324328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:50.361460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:50.417167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 330/637] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:50.417167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21839f31-cfb2-40e9-9313-144aa0bfa527 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:50.513412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:50.519690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e81187d-dfbd-4723-9027-9544bef0557d None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:50.599320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:50.600422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:50.600962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:50.601773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:50.619505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:50.619755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21839f31-cfb2-40e9-9313-144aa0bfa527 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:50.622136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:50.644606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:50.646192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:08:50.647471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:50.651769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 308/638] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:08:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:50.654070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:50.697513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:50.700376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:50.701589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:50.702853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:50.725556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:50.728417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:50.751951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:50.753769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:08:50.755468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:50.761101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 331/639] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:50 2022] GET /identity/v3/users/cinder => generated 84 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:50.762493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:50.827244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:50.891564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0b535f-fbac-4b5c-947c-b6c3877f7f3b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:50.975193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3dc5f4-d1b4-4f10-ac8c-c53ff3807d93 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:50.975607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:50.976844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:50.977673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:50.978370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:50.979911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:50.980542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:50.982866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:51.006243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 309/640] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 348 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:51.006243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:51.040942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:51.042014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:51.042453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:51.042885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:51.044119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:51.044623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:51.045986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:51.047203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:51.077506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 332/641] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:08:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:51.078526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:51.182083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca98a838-0af1-4b13-a36f-892994529b39 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:51.229535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:51.238066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:51.239088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:51.239456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:51.239849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:51.241013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:51.241456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:51.243267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:51.244916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c259e316-e2d3-4947-b68c-c74efbc7e223 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:51.268630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 310/642] 10.222.0.52 () {56 vars in 1137 bytes} [Sat Aug 27 02:08:51 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:51.275740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:51.303717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:51.304709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:51.305113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:51.305511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:51.322671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:51.330672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:51.331740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:51.332645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:51.337495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 333/643] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:51.338991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:51.406851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:51.443789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427de43c-8363-47b5-b980-fbaa40793cbe None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:51.505038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:51.505753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:51.506024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:51.506317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08f0ee5-ccff-4352-862d-f4133b4e3acc None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:51.506423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:51.525543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:51.528395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:51.550620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:51.551821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:51.552953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:51.552953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:51.552953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:51.552953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 311/644] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:51.553629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:51.561831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:51.564757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:51.565539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:51.566172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:51.609378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:51.697937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:51.700266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:51.737023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705f5ac0-7e29-4a8c-b505-e5b203ebc6fd None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:51.792981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:51.802378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:51.803892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:51.804407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:51.804911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:51.884939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 334/645] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:51.885692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:51.945228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:51.946727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:51.968978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:52.057412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448e9625-fda7-4321-b338-c8aaf5d4a4d8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:52.124368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:52.125054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:52.125364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:52.125844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:52.126696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:52.126986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:52.128565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:52.147489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 335/646] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:52.147489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:52.152620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 312/647] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 600 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:52.155169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:52.221703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:52.364011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f105f173-509b-4196-a861-cb473d1b980e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:52.378845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df46f0a0-35fc-48ea-ac93-a76a4ed3f9be None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:52.385442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:52.434970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:52.436148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:52.436751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:52.437451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:52.439134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:52.440310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:52.442464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:52.443266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:52.443585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:52.443752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:52.443880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:52.460849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:52.462834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:52.465397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 336/648] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:52.466677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:52.490934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:52.493403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:52.495618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:52.495618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:52.495618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:52.495618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 313/649] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:52.495618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:52.649650 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:52.670581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdee2044-4c78-4086-b1d2-c9240a5847af None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:52.682338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a190a67-3cd2-4907-8d3f-0a6141e5bd43 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:52.726311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:52.727308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:52.727742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:52.728168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:52.737959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:52.738953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:52.739395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:52.739819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:52.746672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:52.749412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:52.758787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:52.761528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:52.778316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:52.779973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:52.781802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:52.781802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:52.781802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:52.781802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 337/650] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:52.781802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:52.791112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:52.792222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:52.793894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:52.793894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:52.793894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:52.793894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 314/651] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:52.793894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:52.833361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:52.950583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d289e47b-70d0-4da2-a1f0-6f224ff7ce01 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:52.972490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600f1936-f279-41a2-a0dc-fc8efbdf8722 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:53.016646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2fd91e-63cc-45e5-b66f-f18520f1510f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:53.017737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2fd91e-63cc-45e5-b66f-f18520f1510f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.018021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2fd91e-63cc-45e5-b66f-f18520f1510f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.018416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2fd91e-63cc-45e5-b66f-f18520f1510f None admin] PATH_INFO: `/v3/users/glance` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.033713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2fd91e-63cc-45e5-b66f-f18520f1510f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:53.034003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:53.034546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.034915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.035275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.035434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2fd91e-63cc-45e5-b66f-f18520f1510f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:53.036242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:53.036692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:53.038377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:53.041465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-4e2fd91e-63cc-45e5-b66f-f18520f1510f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:53.049956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:53.051633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 338/652] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:52 2022] GET /identity/v3/users/glance => generated 84 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81c04f-2421-45fe-af74-c991f0064c49 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.053364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81c04f-2421-45fe-af74-c991f0064c49 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.054616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81c04f-2421-45fe-af74-c991f0064c49 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.054616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 339/653] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:53.055615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81c04f-2421-45fe-af74-c991f0064c49 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:53.075309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 315/654] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:53.075309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:53.193300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:53.224465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81c04f-2421-45fe-af74-c991f0064c49 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:53.256459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca7a9ce-af46-4252-a34a-e91c5bc01c32 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:53.288763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:53.289665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.290076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.290527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.291530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:53.291931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:53.293812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:53.318861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 340/655] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:53.318861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:53.319147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe191ba-af1c-4c1d-952f-734f02181f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:53.319147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe191ba-af1c-4c1d-952f-734f02181f60 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.319147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe191ba-af1c-4c1d-952f-734f02181f60 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.319147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe191ba-af1c-4c1d-952f-734f02181f60 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.319473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe191ba-af1c-4c1d-952f-734f02181f60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:53.319473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe191ba-af1c-4c1d-952f-734f02181f60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:53.320150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe191ba-af1c-4c1d-952f-734f02181f60 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:53.334483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 316/656] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:53.335799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3b3fcf-0c76-4f49-91dc-5df7a3eebba2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.336159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3b3fcf-0c76-4f49-91dc-5df7a3eebba2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.336506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3b3fcf-0c76-4f49-91dc-5df7a3eebba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.359008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-de3b3fcf-0c76-4f49-91dc-5df7a3eebba2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:53.360461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-de3b3fcf-0c76-4f49-91dc-5df7a3eebba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:53.441945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:53.473143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c75af6-5eed-4410-8c12-1e449d0d3937 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:53.476868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 317/657] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:53.477967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3b3fcf-0c76-4f49-91dc-5df7a3eebba2 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:53.549945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:53.551027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.551473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.551825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.552613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:53.552942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:53.554610 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:53.593875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 341/658] 10.222.0.52 () {56 vars in 1137 bytes} [Sat Aug 27 02:08:53 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:53.593875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:53.599118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:53.678821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3b3fcf-0c76-4f49-91dc-5df7a3eebba2 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:53.780418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:53.781638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.782125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.782580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.798265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:53.802337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:53.817541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91ba47-bf82-4ee2-a536-74bcec468dcd None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:53.822674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:53.823653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 318/659] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:53.830011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:53.882887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:53.883535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:53.883804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:53.884076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:53.892446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:53.893938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:53.905534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:53.913869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:53.915871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 342/660] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:53 2022] GET /identity/v3/users/neutron => generated 85 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:53.917700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:53.994676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b60d3b-fad1-4cc6-be1f-143ee7b93006 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.017482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:54.052966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.053866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.054282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.054723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.068597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.071222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.073097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2253b672-0c2a-4625-809c-84ab81cb5703 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:54.095199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:54.096176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:54.097071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:54.097610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 319/661] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:54.098746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.118193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.119153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.119492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.119764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.218979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.220492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.254608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98631a00-5942-446f-a1e1-4f5e607edfe6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.264693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:54.298728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:54.306843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.307163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.307442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.307719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.320335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:54.325544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:54.327084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 320/662] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:54.328180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.359357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 343/663] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:54.359954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.470556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc46904-c744-40b9-b81e-302dbbb2e242 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.499417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e4e89-1107-4463-8b9c-b720a9479d71 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.508447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.509054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.509422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.509693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.510425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:54.510730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.512200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.526570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 321/664] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:54.527454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.541088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.541833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.542101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.542362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.543114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:54.543411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.545159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.561779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:54.571629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 344/665] 10.222.0.52 () {56 vars in 1139 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:54.571629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.596347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:54.669540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cee5733-ce23-4b92-84ed-d0c3c23be5f1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.694844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c753aa-561c-4bbf-bb3e-d227da580c36 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.717378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.718446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.718922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.719401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.720811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:54.721357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.723471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.743222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 322/666] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:54.744173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.745721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.746696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.746981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.747253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.748181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:54.748486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.750071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.763199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 345/667] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:54.764848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.871014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:54.891778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26a3b54-ffdc-4f34-8b14-932106b13890 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.895508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:54.925750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab932261-0bdb-4aa6-ab5d-49d9a1b49440 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:54.942818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.943785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.944238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.944651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.961346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.963243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.976371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:54.977198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:54.977517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:54.977791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:54.978571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=c83f0d0e2e8047a98fbe114ceb15c6e0) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:54.978988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:54.980628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:54.982008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:54.983821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:54.986481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:54.986481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:54.986481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:54.986481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 323/668] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:54.986481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:54.998528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 346/669] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/role_assignments?user.id=c83f0d0e2e8047a98fbe114ceb15c6e0&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 608 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:55.000556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:55.166024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ac09cf-ce95-492e-9df3-77dce57d479b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:55.173913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42882a76-d94b-4522-af9e-79e5ae8086c5 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:55.231854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:55.232860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:55.233476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:55.234012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:55.246838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:55.247463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:55.247817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:55.248360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:55.248954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:55.250737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=598bc689672741c0ac600e42cf45e39d) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:55.251259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:55.253932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:55.264320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 347/670] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/role_assignments?user.id=598bc689672741c0ac600e42cf45e39d&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 608 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:55.265737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:55.304120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:55.349062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:55.350711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:55.476380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33c02b-8503-4f59-b7da-06e3aa67590d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:55.502972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 324/671] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 519 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:55.504403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:55.541335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:55.542123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:55.542439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:55.542589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:55.557600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:55.559370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:55.577685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:55.591909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:55.595099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:55.596678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:55.596678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:55.596678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:55.596678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 348/672] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:55.596678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:55.691643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ffc10c-bdc6-4a55-91f8-114b1f9a8d83 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:55.702201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:55.762234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:55.763162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:55.764885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:55.765849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:55.767154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:55.767525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:55.770281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:55.774166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ec0606-880b-44e4-9ee7-d1fc7a1d623d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:55.792915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 325/673] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:55.794161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:55.863996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:55.865375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:55.865820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:55.866250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:55.890357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:55.893406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:55.924345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:55.926844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:55.928370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:55.928370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:55.928370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:55.937159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 349/674] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:55.937159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:56.006779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:56.017432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd045a-d03c-4741-8336-01321bd16c9c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:56.079438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:56.080472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:56.080948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:56.081479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:56.082611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:56.083142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:56.085728 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:56.110644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 326/675] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:56.112209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:56.179195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b63145-d851-423b-a327-e68a7d3ab27f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:56.208507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:56.244934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:56.246263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:56.246766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:56.247230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:56.258289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:56.260489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:56.280692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:56.283012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:56.284714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:56.294033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 350/676] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 356 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:56.294033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:56.328471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a87fdd-8beb-408a-a094-2549a5044a04 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:56.378269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:56.378882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:56.379197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:56.379611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:56.380433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:56.380747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:56.382291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:56.398286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 327/677] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:56.398751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:56.432603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:56.543808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a81409-c550-456a-acb6-ae7e3d2d9487 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:56.554184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716c0b36-5d8d-46c6-83b9-9a732d3022a9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:56.605022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:56.605731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:56.606005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:56.606337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:56.615770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:56.617245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:56.617792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:56.619106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:56.619894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:56.620367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:56.622523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:56.623076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:56.625514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:08:56.632065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:56.633579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 351/678] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:08:56 2022] GET /identity/v3/users/swift => generated 83 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:56.635507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:56.662012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 328/679] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:56.664452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:56.702498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:56.798155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:56.807188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9007b71b-1333-4629-95ce-e49423caded4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:56.837102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc86dc65-ffbc-4499-b561-07e737b9f655 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:56.879623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:56.880829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:56.881373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:56.881858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:56.895194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:56.895691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:56.896531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:56.897007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:56.897662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:56.897998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:56.899055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:56.899566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:56.901633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:56.921620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:56.922816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:56.923697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:56.924676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 352/680] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:56.925784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:56.928427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 329/681] 10.222.0.52 () {56 vars in 1135 bytes} [Sat Aug 27 02:08:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:56.936733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:57.079660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281e7299-c867-41d3-9d34-8be04da34c5b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:57.095420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:57.104136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee3fc49-5642-4ca1-af40-e70db50f3955 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:57.132925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:57.133578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:57.133856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:57.134270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:57.135093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:57.135401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:57.137688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:57.141035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:57.148696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:57.149732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:57.150187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:57.150747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:57.151661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=ab71b242a16641429c29dc8aa67867e5) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:57.152141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:57.160148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:57.160536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 330/682] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:57.160536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:57.172616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 353/683] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:56 2022] GET /identity/v3/role_assignments?user.id=ab71b242a16641429c29dc8aa67867e5&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 608 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:57.174453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:57.328509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bbe515-5a13-4b83-9456-9cbb73ae9296 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:57.355185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ece2a7-4881-4a81-afe9-1e952b1ea73a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:57.395865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14acdbc0-ce86-4e76-b11f-db8124761f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:57.396672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14acdbc0-ce86-4e76-b11f-db8124761f4f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:57.397078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14acdbc0-ce86-4e76-b11f-db8124761f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:57.397385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14acdbc0-ce86-4e76-b11f-db8124761f4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:57.436184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:57.437049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:57.437519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:57.437866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:57.438832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=56aaa029a9c542e0bc4658c5cd55c2b9) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:57.439256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:57.441229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:57.462317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 354/684] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:57 2022] GET /identity/v3/role_assignments?user.id=56aaa029a9c542e0bc4658c5cd55c2b9&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 608 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:57.529319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:57.546177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14acdbc0-ce86-4e76-b11f-db8124761f4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:57.549264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14acdbc0-ce86-4e76-b11f-db8124761f4f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:57.740754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 331/685] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:08:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 586 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:08:57.747102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:57.804447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:57.843815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb64d9e-5b4e-4f93-bae5-6edb07712ec7 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:57.844267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb64d9e-5b4e-4f93-bae5-6edb07712ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:57.844736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb64d9e-5b4e-4f93-bae5-6edb07712ec7 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:57.855903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 332/686] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:57 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:57.876247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f92e528-5a0e-451c-920f-179e01c26c80 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:57.876691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f92e528-5a0e-451c-920f-179e01c26c80 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:57.877181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f92e528-5a0e-451c-920f-179e01c26c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:57.929847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7f92e528-5a0e-451c-920f-179e01c26c80 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:57.931780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7f92e528-5a0e-451c-920f-179e01c26c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:58.017632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ddc043-c07f-4c46-b752-b9ac9620a68a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:58.075050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 333/687] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:08:58.076408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f511f88-25ba-428e-acc7-89dd359fd875 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:58.076686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f511f88-25ba-428e-acc7-89dd359fd875 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:58.076952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f511f88-25ba-428e-acc7-89dd359fd875 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:58.083221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 334/688] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:08:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:08:58.084597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cee520-0136-43f5-92c7-46f208ead967 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:58.084870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cee520-0136-43f5-92c7-46f208ead967 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:58.085160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cee520-0136-43f5-92c7-46f208ead967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:58.113492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f8cee520-0136-43f5-92c7-46f208ead967 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:08:58.114715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f8cee520-0136-43f5-92c7-46f208ead967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:08:58.118229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:58.118884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:58.119162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:58.119431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:58.143655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:58.145145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:58.161468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:58.184808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:58.185875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:58.187125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:58.187125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:58.187125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:58.187125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 355/689] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 440 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:58.193059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:58.304023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 335/690] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:08:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:08:58.331969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:58.365579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b4807d-9e41-4053-b11a-40ae0df5930a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:58.411356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:58.412290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:58.412654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:58.412993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:58.413800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:58.414247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:58.416570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:58.441269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 356/691] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:58.446287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cee520-0136-43f5-92c7-46f208ead967 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:58.472170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:58.670558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cee520-0136-43f5-92c7-46f208ead967 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:58.692743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd7032-9ac2-4a20-8192-634095ff187a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:58.770427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:58.771351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:58.771769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:58.772290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:58.777458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:58.779238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:58.779912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:58.780569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:58.786340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:58.786558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:58.790126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:58.801219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:08:58.808330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:58.809743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 336/692] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:08:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 364 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:58.811215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:08:58.815483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:58.816955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:08:58.818512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:58.823056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 357/693] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:08:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:58.824865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:58.845803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:58.970733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae150ce-502f-4db2-800f-81b9b6b21430 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:59.036736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:59.037544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:59.037816 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:59.038082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:59.038738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:59.039586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad457f1-632e-4be0-91dd-b615425868ef None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:59.040123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:59.056002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:59.058033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:08:59.059583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:08:59.065381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 337/694] 10.222.0.52 () {56 vars in 1129 bytes} [Sat Aug 27 02:08:58 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:59.066421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:59.108356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:59.109806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:59.110266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:59.110724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:59.111952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:59.112469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:59.115556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:59.138530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 358/695] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:08:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:59.140117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:59.179760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:59.207324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ee480-556f-4a97-9865-1c04e09636e3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:59.246828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:59.261377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:59.262267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:59.262686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:59.263094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:59.263970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:59.264418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:59.266528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:59.285477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 338/696] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:08:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:59.286961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:59.334176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93751737-6985-4d4b-a22e-6584118c1621 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:59.418316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:59.419412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:59.419966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:59.420442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:59.421730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:59.422270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:59.424803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:59.441160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 359/697] 10.222.0.52 () {56 vars in 1124 bytes} [Sat Aug 27 02:08:59 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:59.441160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:59.519385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86326a64-387c-4144-b3a4-8f8b82302af2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:59.572761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:59.573917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:59.574391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87edc48a-f255-4792-8373-9c4f8a965c13 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:59.574645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:59.575143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:59.576405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=cfc9a334f972474aa19aa3beae51560f) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:59.578341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:59.580190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:59.590018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 339/698] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:08:59 2022] GET /identity/v3/role_assignments?user.id=cfc9a334f972474aa19aa3beae51560f&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 608 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:59.591904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:59.605390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:59.612255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:08:59.651297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:59.652203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:59.657444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:59.657901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:59.702546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:59.704509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:08:59.741597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:08:59.742663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:08:59.743872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:08:59.743872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:08:59.743872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:08:59.743872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 360/699] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:08:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:08:59.759311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:08:59.852219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24229a2f-3d0c-473f-ab1d-eb12d1bf95c4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:08:59.915699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:08:59.917446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:08:59.917992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:08:59.918974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:08:59.920029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:08:59.920894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:08:59.923233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:08:59.937175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 340/700] 10.222.0.52 () {56 vars in 1124 bytes} [Sat Aug 27 02:08:59 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:08:59.953903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2edf9b3-c3d9-424d-b89d-1c0ca02a3976 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:00.015973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58904a54-2998-40b7-87ed-a606b6189764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:00.016934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58904a54-2998-40b7-87ed-a606b6189764 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.017396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58904a54-2998-40b7-87ed-a606b6189764 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.017835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58904a54-2998-40b7-87ed-a606b6189764 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.018907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58904a54-2998-40b7-87ed-a606b6189764 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:00.019394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58904a54-2998-40b7-87ed-a606b6189764 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:00.022015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58904a54-2998-40b7-87ed-a606b6189764 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:00.046030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 361/701] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:08:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:00.054300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:00.158147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:00.190385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739d4fc-8d98-4a96-9c7c-3179d3127bd3 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.191287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739d4fc-8d98-4a96-9c7c-3179d3127bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.191707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739d4fc-8d98-4a96-9c7c-3179d3127bd3 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.194846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 362/702] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:00.203782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434a60b0-9160-467a-9da9-251b73e16731 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.204398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434a60b0-9160-467a-9da9-251b73e16731 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.204505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ffca38-f2a7-4aad-9df7-ab8d377567fa None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:00.205084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434a60b0-9160-467a-9da9-251b73e16731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.239464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-434a60b0-9160-467a-9da9-251b73e16731 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:00.241563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-434a60b0-9160-467a-9da9-251b73e16731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:00.263660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:00.265008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.265454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.265828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.284505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:00.287796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:00.305264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:00.306321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:00.307517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:00.310218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 341/703] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:09:00 2022] GET /identity/v3/users/nova => generated 82 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:00.317787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:00.328309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:00.459488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 363/704] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:09:00.461209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e438f803-3dd6-4e29-8e18-57e72b77f170 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.461337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e438f803-3dd6-4e29-8e18-57e72b77f170 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.461665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e438f803-3dd6-4e29-8e18-57e72b77f170 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.465717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820003a8-e4dd-4a73-8b7d-4816c3c37de6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:00.469456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 364/705] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:00.473339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08cbf57-5952-418e-9a32-7f882d63708e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.473855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08cbf57-5952-418e-9a32-7f882d63708e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.474271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08cbf57-5952-418e-9a32-7f882d63708e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.508563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f08cbf57-5952-418e-9a32-7f882d63708e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:00.510622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f08cbf57-5952-418e-9a32-7f882d63708e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:00.513584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:00.521444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:00.522254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.522639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.523077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.524230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:00.524981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:00.526545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:00.555872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 342/706] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:09:00 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:00.566063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:00.695780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 365/707] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:09:00.810167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb91e69-31ef-4baa-90e4-2931f1ca934c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:00.867586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:00.868447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:00.868758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:00.869028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:00.954240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08cbf57-5952-418e-9a32-7f882d63708e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:00.967834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:00.986422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:00.988848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.111498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:01.123605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08cbf57-5952-418e-9a32-7f882d63708e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:01.160698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 343/708] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 595 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 27 02:09:01.169941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.172215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:01.173737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:01.174557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:01.175424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] PATH_INFO: `/v3/services/network` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:01.176820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:01.178630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:01.193766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:01.194804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:01.195626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:01.199762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 366/709] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:00 2022] GET /identity/v3/services/network => generated 88 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:01.208533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.323573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:01.344974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd2333-abed-4cb0-bf1b-74e1a50a2822 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:01.403682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283f5db2-6c7c-42c2-8147-1b392c145365 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:01.410845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:01.411876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:01.412381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:01.412845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:01.427567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:01.429672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:01.450852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:01.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:01.453383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:01.453383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:01.453383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:01.455826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 344/710] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:01.456962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.481102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:01.483138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:01.483138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:01.483138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:01.483547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] RBAC: Adding query filter params (name=network) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:01.484119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:01.486193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.492131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 367/711] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:01 2022] GET /identity/v3/services?name=network => generated 124 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:01.493172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.518206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:01.587052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f4f0e9-d61a-4b61-a95d-2e2a6de7ae8e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:01.609466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db586a01-0e7c-4f0e-bda8-35d25f3580ce None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:01.624870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:01.625968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:01.626463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:01.626937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:01.636949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:01.641390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:01.643202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.651682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:01.652298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:01.652599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:01.652925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:01.654099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:01.655879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:01.657506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:01.661766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:01.662869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:01.663742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:01.663742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:01.663742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:01.666304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 345/712] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:01.667584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.680255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 368/713] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:01.681952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.819599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:01.828738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2927b-a65a-48d6-9a5e-9f165e841387 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:01.844447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae313b0d-8ddf-4745-b6c5-4c712f63c237 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:01.869444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:01.870865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:01.871446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:01.871912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:01.872934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] RBAC: Adding query filter params (type=network) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:01.873488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:01.876270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.885162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 346/714] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:01 2022] GET /identity/v3/services?type=network => generated 124 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:01.886838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.897305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:01.898249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:01.898663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:01.899078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:01.900106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:01.900570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:01.903330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:01.935952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 369/715] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:01.942776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:01.966846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:02.018092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da018dfa-f4b0-4bfd-b3c7-11f72a07ad2b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:02.066080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:02.069427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:02.070025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:02.070496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:02.086806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:02.090451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:02.125487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:02.126949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:02.129077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 347/716] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:02.136861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:02.141357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da4b837-d220-43ea-808d-460e63811589 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:02.197139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:02.198163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:02.198791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:02.199259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:02.200281 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:02.203012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:02.227475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:02.242538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 370/717] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:09:01 2022] POST /identity/v3/users => generated 254 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:02.319486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19492aff-5ff9-4dfa-86cf-93b36c8f279b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:02.343261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:02.368800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f05bc0-55f8-4d5b-84be-ec9b4e92f9e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:02.370387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05bc0-55f8-4d5b-84be-ec9b4e92f9e9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:02.371085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05bc0-55f8-4d5b-84be-ec9b4e92f9e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:02.371808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05bc0-55f8-4d5b-84be-ec9b4e92f9e9 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:02.373038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05bc0-55f8-4d5b-84be-ec9b4e92f9e9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:02.373752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05bc0-55f8-4d5b-84be-ec9b4e92f9e9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:02.376303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05bc0-55f8-4d5b-84be-ec9b4e92f9e9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:02.393675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 348/718] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:02.402028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:02.556025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30f275-ef2c-4e35-a0fc-784f15c7d85c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:02.603168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a17f50-982f-4b98-bd1a-0ed960eb7adf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:02.604627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a17f50-982f-4b98-bd1a-0ed960eb7adf None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:02.605485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a17f50-982f-4b98-bd1a-0ed960eb7adf None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:02.606193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a17f50-982f-4b98-bd1a-0ed960eb7adf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:02.607271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a17f50-982f-4b98-bd1a-0ed960eb7adf None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=c83f0d0e2e8047a98fbe114ceb15c6e0) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:02.607786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a17f50-982f-4b98-bd1a-0ed960eb7adf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:02.610176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a17f50-982f-4b98-bd1a-0ed960eb7adf None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:02.618041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 371/719] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:02 2022] GET /identity/v3/role_assignments?user.id=c83f0d0e2e8047a98fbe114ceb15c6e0&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:02.923730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5fe581-9318-4b29-acc1-466255892962 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:02.924263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5fe581-9318-4b29-acc1-466255892962 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:02.924565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5fe581-9318-4b29-acc1-466255892962 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:02.926208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 349/720] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:02.932785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2be027-5603-4df6-931c-a7db5f076215 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:02.933094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2be027-5603-4df6-931c-a7db5f076215 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:02.933377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2be027-5603-4df6-931c-a7db5f076215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:02.963067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5d2be027-5603-4df6-931c-a7db5f076215 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:02.964365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5d2be027-5603-4df6-931c-a7db5f076215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:03.016220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce713eb-3d0c-4f51-94d0-12356ffca736 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:03.016773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce713eb-3d0c-4f51-94d0-12356ffca736 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:03.017594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce713eb-3d0c-4f51-94d0-12356ffca736 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:03.025544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 350/721] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:03 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:03.043179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e76cd60-6d70-475b-a812-1ac7e09aa094 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:03.043646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e76cd60-6d70-475b-a812-1ac7e09aa094 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:03.044076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e76cd60-6d70-475b-a812-1ac7e09aa094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:03.062472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:03.083627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 372/722] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:09:03.092175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2e76cd60-6d70-475b-a812-1ac7e09aa094 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:03.094182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2e76cd60-6d70-475b-a812-1ac7e09aa094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:03.236570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 351/723] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:09:03.430941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2be027-5603-4df6-931c-a7db5f076215 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:03.608613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2be027-5603-4df6-931c-a7db5f076215 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:03.705909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:03.707513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:03.708208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:03.708795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:03.710873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:03.714049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:03.754885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 373/724] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:03 2022] POST /identity/v3/services => generated 251 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:03.854843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e76cd60-6d70-475b-a812-1ac7e09aa094 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:04.031654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e76cd60-6d70-475b-a812-1ac7e09aa094 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:04.041799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:04.114930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:04.116211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:04.116774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:04.117341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:04.139213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:04.148464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:04.149725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:04.151182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 352/725] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:04.159581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:04.254613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:04.346016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c5b443-a986-4d15-a28f-7aab45aa8160 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:04.400511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:04.401198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:04.401699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:04.402020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:04.402895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:04.403206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:04.404666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:04.419113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 374/726] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:09:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:04.428184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:04.628759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f93419-1626-4ba9-8f08-64d6296f36f2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:04.695178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:04.696205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:04.696683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:04.697280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:04.716935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:04.719773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:04.745162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:04.747603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:04.751192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:04.751192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:04.751192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:04.751192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 353/727] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:04.753958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:04.962963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e476266-de43-484b-9dc3-190b9fbb48ff None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:05.001681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:05.015814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:05.017074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.017491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.018153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.018975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:05.019497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:05.021387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:05.035648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 375/728] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:05.041382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:05.060359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:05.168538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd685b-6b5a-4fb0-93b3-683e4a563948 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:05.215385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9179e8-0cdd-40a6-bbbe-8b6226e003dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:05.216353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9179e8-0cdd-40a6-bbbe-8b6226e003dc None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.216818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9179e8-0cdd-40a6-bbbe-8b6226e003dc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.217327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9179e8-0cdd-40a6-bbbe-8b6226e003dc None admin] PATH_INFO: `/v3/users/swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.231646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9179e8-0cdd-40a6-bbbe-8b6226e003dc None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:05.234286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9179e8-0cdd-40a6-bbbe-8b6226e003dc None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-5f9179e8-0cdd-40a6-bbbe-8b6226e003dc None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:05.252057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:05.253075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:09:05.254052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:05.255133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 354/729] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:05 2022] GET /identity/v3/users/swift => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:05.261015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:05.420383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a3ae19-ae30-4c97-bc4d-21efc51de92e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:05.452922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be55f37a-ffe4-439b-92e6-856363a98a32 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.453418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be55f37a-ffe4-439b-92e6-856363a98a32 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.453856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be55f37a-ffe4-439b-92e6-856363a98a32 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.456484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 355/730] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:05.465992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f521746-63a4-4970-821d-8fa8caeb7191 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.466422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f521746-63a4-4970-821d-8fa8caeb7191 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.466739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f521746-63a4-4970-821d-8fa8caeb7191 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.496322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-6f521746-63a4-4970-821d-8fa8caeb7191 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:05.497795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6f521746-63a4-4970-821d-8fa8caeb7191 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:05.505922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:05.506957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.507722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.508201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.509311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:05.509846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:05.511559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:05.542523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 376/731] 10.222.0.52 () {56 vars in 1135 bytes} [Sat Aug 27 02:09:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:05.550823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:05.604944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 356/732] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:05 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 27 02:09:05.642746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:05.764269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111ff56-fb44-4236-8dec-634da9887393 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.764539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111ff56-fb44-4236-8dec-634da9887393 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.764803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111ff56-fb44-4236-8dec-634da9887393 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.766593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 357/733] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:05.776865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4063752-5ca9-4901-8676-dd61674bb75b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.777187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4063752-5ca9-4901-8676-dd61674bb75b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.777441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4063752-5ca9-4901-8676-dd61674bb75b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.793077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff389f-c2f4-49d0-b365-2dfb5d7da537 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:05.805605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:05.817158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-d4063752-5ca9-4901-8676-dd61674bb75b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:05.826698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d4063752-5ca9-4901-8676-dd61674bb75b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:05.837573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91af4963-a091-4092-90d2-13b9166a1ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:05.838192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91af4963-a091-4092-90d2-13b9166a1ab6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:05.838471 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91af4963-a091-4092-90d2-13b9166a1ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:05.838747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91af4963-a091-4092-90d2-13b9166a1ab6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:05.932560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91af4963-a091-4092-90d2-13b9166a1ab6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:05.934371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91af4963-a091-4092-90d2-13b9166a1ab6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:05.973690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:06.045801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 358/734] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:05 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 27 02:09:06.047972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4063752-5ca9-4901-8676-dd61674bb75b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:06.068560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 377/735] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:05 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 27 02:09:06.068560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc6c55-5d6e-4b94-a325-8ec17c9454ac None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.068560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc6c55-5d6e-4b94-a325-8ec17c9454ac None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.068560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc6c55-5d6e-4b94-a325-8ec17c9454ac None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.068560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 378/736] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:06.069677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bc6c55-5d6e-4b94-a325-8ec17c9454ac None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:06.204730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bc6c55-5d6e-4b94-a325-8ec17c9454ac None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:06.208044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4063752-5ca9-4901-8676-dd61674bb75b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:06.241560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:06.249077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57290da2-f8a4-4914-bb0d-5acd3a3fdf88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:06.249854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57290da2-f8a4-4914-bb0d-5acd3a3fdf88 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.250195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57290da2-f8a4-4914-bb0d-5acd3a3fdf88 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.250379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57290da2-f8a4-4914-bb0d-5acd3a3fdf88 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.262110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57290da2-f8a4-4914-bb0d-5acd3a3fdf88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:06.263637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57290da2-f8a4-4914-bb0d-5acd3a3fdf88 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:06.275900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:06.276926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.277482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.277973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.278865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:06.280991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-57290da2-f8a4-4914-bb0d-5acd3a3fdf88 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:06.282468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:06.283405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:06.284386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:06.284386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:06.284386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:06.289006 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:06.289197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 379/737] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:06.291434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d752a6-90b2-4903-91bb-780a5dbed529 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.291855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d752a6-90b2-4903-91bb-780a5dbed529 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.292217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d752a6-90b2-4903-91bb-780a5dbed529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.313353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 359/738] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:06 2022] POST /identity/v3/services => generated 240 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:06.315328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:06.315675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-18d752a6-90b2-4903-91bb-780a5dbed529 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:06.316957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-18d752a6-90b2-4903-91bb-780a5dbed529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:06.420083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 380/739] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:06.539732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc55841-0ad0-44fe-9688-c8430f63c507 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:06.602048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:06.602845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.603132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.603389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.604202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:06.604514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:06.609064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:06.616852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d752a6-90b2-4903-91bb-780a5dbed529 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:06.628314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 360/740] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:06.634759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:06.757891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:06.788467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d752a6-90b2-4903-91bb-780a5dbed529 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:06.799727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:06.830428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea0be3-4763-403a-9113-cc88c3532247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:06.831085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea0be3-4763-403a-9113-cc88c3532247 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.831349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5199da-f9f2-49d7-b564-8ddf48aea505 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:06.831504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea0be3-4763-403a-9113-cc88c3532247 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.831645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea0be3-4763-403a-9113-cc88c3532247 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.845068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaea0be3-4763-403a-9113-cc88c3532247 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-eaea0be3-4763-403a-9113-cc88c3532247 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:06.850341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:06.852026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 381/741] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:09:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:06.853520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c707b405-cec8-405e-96a2-e7f1f34b2cbd None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.854842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c707b405-cec8-405e-96a2-e7f1f34b2cbd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.854842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c707b405-cec8-405e-96a2-e7f1f34b2cbd None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.855850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 382/742] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:06.864332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd8303-ac68-457c-bfc2-256bcd5ace70 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.864865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd8303-ac68-457c-bfc2-256bcd5ace70 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.865387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd8303-ac68-457c-bfc2-256bcd5ace70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.874766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:06.875859 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:06.875859 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:06.876292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:06.884281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:06.886770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-0ffd8303-ac68-457c-bfc2-256bcd5ace70 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:06.887038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:06.887986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0ffd8303-ac68-457c-bfc2-256bcd5ace70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:06.904932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:06.908046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:06.911110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:06.922269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 361/743] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:06.922269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:06.991311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 383/744] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:06.992196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffd8303-ac68-457c-bfc2-256bcd5ace70 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.087474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b97f811-4b1a-4aa5-b915-a8b92967b7b2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.102947 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:07.132750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.133855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.134280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.134760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.135779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:07.136191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:07.138522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:07.144599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:07.149723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffd8303-ac68-457c-bfc2-256bcd5ace70 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.155217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 362/745] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:09:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:07.157440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.189617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.190853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.191338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.191749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.192617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:07.193084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:07.194938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:07.210189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 384/746] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:07.211397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.303411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c947dea-9797-425f-b9b0-8514598b5a9a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.357170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.358058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.358470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.358874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.377461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:07.387539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:07.389446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:07.391747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:07.395106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 363/747] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:07.396669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.411576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53304fe3-fbbf-4fb9-8846-b21e639ff2b5 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.453140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.454013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.454271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.454551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.463306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:07.472349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:07.475356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:07.494484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:07.496336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:07.497987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:07.497987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:07.497987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:07.501357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 385/748] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:07.502797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.527134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:07.560761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38339ae0-88db-41b8-a17a-d5810b1b60c0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.598288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.598964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.599244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.599515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.600319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=cfc9a334f972474aa19aa3beae51560f) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:07.600630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:07.602360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:07.617248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 364/749] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/role_assignments?user.id=cfc9a334f972474aa19aa3beae51560f&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:07.617248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.690941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc37b2c-03ea-42e1-af56-6dc5230ea4e8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.755682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.756399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.756816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.757197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.757861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:07.759315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:07.769792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:07.771005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:07.771827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 386/750] 10.222.0.52 () {56 vars in 1129 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:07.772599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.795575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:07.801256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ff9cc-c985-4f8b-a6d8-b8fd93a42a37 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.847025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.848019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.848523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.848990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.849908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:07.850847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:07.852697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:07.864485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 365/751] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:07.865801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:07.889984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:07.926537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea29503-e30d-44be-9222-dbf166dd1426 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:07.968276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:07.969112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:07.969539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:07.969965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:07.970725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:07.971127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:07.972762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:07.989713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 387/752] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:07.990814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.011957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294fd3af-24fe-4719-9aa7-d312186494f3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.069962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.070834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.071257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.071642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.072653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:08.073070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.076608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.086001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 366/753] 10.222.0.52 () {56 vars in 1124 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:08.087438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.132893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:08.170742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137c4437-9d43-4a00-9d67-20efe67e376c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.222721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.223838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.223961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.224314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.227244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:08.238372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7356ec2-8a40-4486-be38-da4d61aaf7ce None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.244572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.246898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:08.272199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:08.273803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:08.274921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:08.274921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:08.274921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:08.280755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 388/754] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:08.280755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.295898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.297387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.297824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.298261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.308939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.310745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:08.326445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:08.327559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:08.328652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:08.330677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 367/755] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:08.336194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.428749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:08.447034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6e9319-8955-464a-87f3-cce8686d4a04 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.468192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a80265-7ca9-4f7a-ba43-ff0af5915807 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.492129 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.493387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.493867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.494333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.495196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:08.495695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.497302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.510249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 368/756] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:08.511453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.526559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268b341-0389-43a0-8a03-924139859972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.527739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268b341-0389-43a0-8a03-924139859972 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.528195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268b341-0389-43a0-8a03-924139859972 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.528738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268b341-0389-43a0-8a03-924139859972 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.530037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268b341-0389-43a0-8a03-924139859972 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:08.530608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268b341-0389-43a0-8a03-924139859972 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.533553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268b341-0389-43a0-8a03-924139859972 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.546286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 389/757] 10.222.0.52 () {56 vars in 1124 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:08.547544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268b341-0389-43a0-8a03-924139859972 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.625104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482aa71e-7b97-4496-bc19-d60c03c441a7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.638284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:08.671614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.672997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.673678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.674154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.675135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:08.675661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.677405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.713323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 369/758] 10.222.0.52 () {56 vars in 1149 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:08.713323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.717245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268b341-0389-43a0-8a03-924139859972 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.725710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:08.797015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.798298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.801425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.802014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] PATH_INFO: `/v3/users/nova` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.817998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.821766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:08.851986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:08.853866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:08.855676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:08.861251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 390/759] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:08.863126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:08.913829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8310e-1fb4-4a28-af35-01881744bc4d None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:08.977429 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:08.978508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:08.979044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:08.979587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:08.980986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] RBAC: Adding query filter params (interface=public, service_id=bd950fddc4f34fb2a62ca4e33fad7590, region_id=RegionOne) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:08.981642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:08.984264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:08.997003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 370/760] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/endpoints?service_id=bd950fddc4f34fb2a62ca4e33fad7590&interface=public®ion_id=RegionOne => generated 194 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:08.998092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:09.037911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785a3f74-3887-498d-ac63-8440fba306fc None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:09.057186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:09.099454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f2b6c-b9ac-4faa-a76c-49ed6d1b79f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:09.100663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f2b6c-b9ac-4faa-a76c-49ed6d1b79f9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:09.101619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f2b6c-b9ac-4faa-a76c-49ed6d1b79f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:09.102216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f2b6c-b9ac-4faa-a76c-49ed6d1b79f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:09.152520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:09.201300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6af2f03-f3e8-4944-b602-74aaa79fbc29 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:09.216912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824f2b6c-b9ac-4faa-a76c-49ed6d1b79f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:09.218403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824f2b6c-b9ac-4faa-a76c-49ed6d1b79f9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:09.259938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:09.260740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:09.261178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:09.261610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:09.262796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:09.263266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:09.265699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:09.286111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 371/761] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:09.286968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:09.383586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 391/762] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:08 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:09:09.389299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5873d52-04fc-4127-8884-5bd67ab3f0e2 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:09.389299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5873d52-04fc-4127-8884-5bd67ab3f0e2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:09.389299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5873d52-04fc-4127-8884-5bd67ab3f0e2 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:09.389299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 392/763] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:09.389299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5873d52-04fc-4127-8884-5bd67ab3f0e2 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:09.411018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:09.440604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df702d57-9401-44b7-8a04-1f6422802d0f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:09.478471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:09.491058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:09.491874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:09.492221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:09.492590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:09.493394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:09.495216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:09.509828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 372/764] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:09 2022] GET /identity/v3/services => generated 772 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:09.512044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:09.569448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5873d52-04fc-4127-8884-5bd67ab3f0e2 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:09.633807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e87ef0-7098-4d97-8a8b-18a91a707248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:09.635044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e87ef0-7098-4d97-8a8b-18a91a707248 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:09.635566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e87ef0-7098-4d97-8a8b-18a91a707248 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:09.636038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e87ef0-7098-4d97-8a8b-18a91a707248 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:09.652082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e87ef0-7098-4d97-8a8b-18a91a707248 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:09.654328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e87ef0-7098-4d97-8a8b-18a91a707248 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-34e87ef0-7098-4d97-8a8b-18a91a707248 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:09.676851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:09.678076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:09.679118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:09.679118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:09.679118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:09.679118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 393/765] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:09.680207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d03260-a28d-4aa8-b515-fb43edab6046 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:09.680468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d03260-a28d-4aa8-b515-fb43edab6046 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:09.680743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d03260-a28d-4aa8-b515-fb43edab6046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:09.707587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-26d03260-a28d-4aa8-b515-fb43edab6046 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:09.708797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-26d03260-a28d-4aa8-b515-fb43edab6046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:09.712102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dba600c-3d10-4f87-8c36-1a95e9077b57 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:09.761508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:09.762330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:09.762606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:09.762943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:09.776266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:09.778014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:09.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:09.798594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:09.800284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:09.800284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:09.800284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:09.802259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 373/766] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:09.803258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:09.837926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:09.847518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 394/767] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:09.862947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d03260-a28d-4aa8-b515-fb43edab6046 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:10.027434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:10.079882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d03260-a28d-4aa8-b515-fb43edab6046 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.146642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:10.148077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.148934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.149724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.151272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:10.152054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:10.154763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:10.177885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 395/768] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:10.183871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:10.220559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2675e1-8cbf-422c-948d-63e4fdf8550d None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.269615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de01ec1-0ce1-4d27-bf3e-29e45696a3c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:10.270233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de01ec1-0ce1-4d27-bf3e-29e45696a3c5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.270507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de01ec1-0ce1-4d27-bf3e-29e45696a3c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.270773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de01ec1-0ce1-4d27-bf3e-29e45696a3c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.271462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de01ec1-0ce1-4d27-bf3e-29e45696a3c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:10.271741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de01ec1-0ce1-4d27-bf3e-29e45696a3c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:10.273112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de01ec1-0ce1-4d27-bf3e-29e45696a3c5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:10.277566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:10.289953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 374/769] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 487 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:10.291373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a04dc6-79c2-40bb-8a7f-66b75e1ae4c3 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.291655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a04dc6-79c2-40bb-8a7f-66b75e1ae4c3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.291922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a04dc6-79c2-40bb-8a7f-66b75e1ae4c3 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.293397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 375/770] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:10.293916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a04dc6-79c2-40bb-8a7f-66b75e1ae4c3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:10.348418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d96bcb9-a1b3-43ed-82fd-07a808c1adc2 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.400764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c954dd-1f25-4d90-be7c-01f667e14e24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:10.401486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c954dd-1f25-4d90-be7c-01f667e14e24 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.401762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c954dd-1f25-4d90-be7c-01f667e14e24 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.402079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c954dd-1f25-4d90-be7c-01f667e14e24 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.410279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c954dd-1f25-4d90-be7c-01f667e14e24 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:10.412051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c954dd-1f25-4d90-be7c-01f667e14e24 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:10.413579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a04dc6-79c2-40bb-8a7f-66b75e1ae4c3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.415862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-e9c954dd-1f25-4d90-be7c-01f667e14e24 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:10.426788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:10.427933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:10.429057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 396/771] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:10.429585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100154a2-16cc-4d4f-8b4b-48e26052e7bd None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.429585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100154a2-16cc-4d4f-8b4b-48e26052e7bd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.430208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100154a2-16cc-4d4f-8b4b-48e26052e7bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.449541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-100154a2-16cc-4d4f-8b4b-48e26052e7bd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:10.450746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-100154a2-16cc-4d4f-8b4b-48e26052e7bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:10.465955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:10.467119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.467468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.467991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.480274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:10.482983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:10.504432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:10.505616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:10.506574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 376/772] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:10.507828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:10.533965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 397/773] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:10 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:10.534562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100154a2-16cc-4d4f-8b4b-48e26052e7bd None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:10.603994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:10.691179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100154a2-16cc-4d4f-8b4b-48e26052e7bd None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.700792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331efed8-eef4-42ac-971d-acbd82f5ca94 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.742998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:10.744054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.744644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.745089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] PATH_INFO: `/v3/services/network` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.746381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:10.748903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:10.769776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:10.771631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:10.773247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:10.776790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 377/774] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:10 2022] GET /identity/v3/services/network => generated 88 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:10.778061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:10.778393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:10.779792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:10.780618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:10.781506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:10.783143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:10.784120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:10.787390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:10.814214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 398/775] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:10.816678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:10.827661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:10.953691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdebb6d-c5de-471c-b522-763f16f2006a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.982044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699a9575-a424-47da-82e9-262003dc0560 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:10.985564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:11.029296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.030934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.031776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.032269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.033696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:11.034288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:11.036885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:11.040597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.041558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.041827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.042106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.042970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] RBAC: Adding query filter params (name=network) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:11.043149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:11.044705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:11.050573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 399/776] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:10 2022] GET /identity/v3/services?name=network => generated 124 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:11.051308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:11.067358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 378/777] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:11.068693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:11.181604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:11.206419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d834375-df2d-42c5-8295-633ec46ee792 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:11.220887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f1133-a49b-42e2-8b93-338355138934 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:11.262306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.263836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.264342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.265334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.283430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.284372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.284700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.285076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.286200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:11.286654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:11.287945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:11.288826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:11.298157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 400/778] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:11 2022] GET /identity/v3/role_assignments?user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 259 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:11.299140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:11.305861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:11.306962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:11.307853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:11.310761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 379/779] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:11.312181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:11.349916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:11.458979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efee365-cc47-42c0-bacd-fa3377834504 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:11.497974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be408b-db6f-4d76-9bf2-52bf71b0ab15 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:11.506120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.507148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.507632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.508108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.508965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] RBAC: Adding query filter params (type=network) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:11.509556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:11.511172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:11.519603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 401/780] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:11 2022] GET /identity/v3/services?type=network => generated 350 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:11.521224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:11.560147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.561710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.561710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.562195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/c83f0d0e2e8047a98fbe114ceb15c6e0/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.573414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=c83f0d0e2e8047a98fbe114ceb15c6e0, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:11.573414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:11.596880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 380/781] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:09:11 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/c83f0d0e2e8047a98fbe114ceb15c6e0/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:09:11.598560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:11.617901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:11.797758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:11.820758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d082a2-87c9-4f30-b036-1650ef74ffc6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:11.833269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1160f8-e282-44ad-877f-e99b3db955a1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:11.907201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.908239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.908699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.909186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:11.909339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.910171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:11.910544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] RBAC: Adding query filter params (interface=public, service_id=57b08b10b0f546e093a48bce007ba132, region_id=RegionOne) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:11.910997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:11.911114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:11.911592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:11.921180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:11.921428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:11.922600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:11.924333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 381/782] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:09:11 2022] GET /identity/v3/endpoints?service_id=57b08b10b0f546e093a48bce007ba132&interface=public®ion_id=RegionOne => generated 194 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:11.925802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:11.934325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:11.948264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 402/783] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:09:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 428 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:11.958533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:12.088206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:12.127766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a98d4-80da-4685-82a1-cd05fc9c3bc9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:12.166499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eceb5e7-c5b6-4baf-bf98-fd071d1128fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:12.167143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eceb5e7-c5b6-4baf-bf98-fd071d1128fb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:12.167416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eceb5e7-c5b6-4baf-bf98-fd071d1128fb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:12.167680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eceb5e7-c5b6-4baf-bf98-fd071d1128fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:12.179591 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eceb5e7-c5b6-4baf-bf98-fd071d1128fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:12.181264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eceb5e7-c5b6-4baf-bf98-fd071d1128fb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:12.197564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb2e07a-f08b-496e-bf25-f13daa28d77f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-0eceb5e7-c5b6-4baf-bf98-fd071d1128fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:12.197790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:12.198712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:12.199758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:12.199758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:12.199758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:12.199758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 403/784] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:12.200571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e656-282b-486e-9128-75c388db73eb None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:12.200897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e656-282b-486e-9128-75c388db73eb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:12.206173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb1e656-282b-486e-9128-75c388db73eb None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:12.211971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 404/785] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:12 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:12.211971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e656-282b-486e-9128-75c388db73eb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:12.233171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:12.259380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6309224-c599-4413-b61b-87176f1dcb2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:12.260094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6309224-c599-4413-b61b-87176f1dcb2f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:12.260366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6309224-c599-4413-b61b-87176f1dcb2f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:12.260789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6309224-c599-4413-b61b-87176f1dcb2f None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:12.261483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6309224-c599-4413-b61b-87176f1dcb2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:12.263289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6309224-c599-4413-b61b-87176f1dcb2f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:12.271434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 382/786] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:11 2022] GET /identity/v3/services => generated 772 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:12.272848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6942c240-539c-4d23-b8f9-02a4c607f31d None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:12.273152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6942c240-539c-4d23-b8f9-02a4c607f31d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:12.282034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6942c240-539c-4d23-b8f9-02a4c607f31d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:12.351166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-6942c240-539c-4d23-b8f9-02a4c607f31d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:12.352062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb1e656-282b-486e-9128-75c388db73eb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:12.354512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6942c240-539c-4d23-b8f9-02a4c607f31d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:12.408892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:12.409585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:12.409864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:12.410136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:12.411260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:12.411664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:12.413297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:12.431612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 405/787] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:12.439652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:12.485973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 383/788] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:12.565987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:12.604940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1df0f6-81ed-4892-9da2-f553af2d78a8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:12.646062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:12.646728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:12.647008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:12.647281 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] PATH_INFO: `/v3/users/swift` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:12.656298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:12.657905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:12.673322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:12.674348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:12.675317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 406/789] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:12 2022] GET /identity/v3/users/swift => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:12.685735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6942c240-539c-4d23-b8f9-02a4c607f31d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:12.803560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:12.833624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:12.885804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6942c240-539c-4d23-b8f9-02a4c607f31d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:12.928982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:12.933159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:12.933159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:12.933159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:12.933159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:12.933159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:12.933159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:12.956047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 384/790] 10.222.0.52 () {56 vars in 1135 bytes} [Sat Aug 27 02:09:12 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:12.962531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:13.049498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ba1bd-9672-42c6-9875-2ed6133c54ac None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:13.086845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:13.097975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:13.099233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:13.099750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:13.100227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:13.101054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:13.106298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:13.127693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:13.128801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 27 02:09:13.129686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:13.140877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 407/791] 10.222.0.52 () {56 vars in 1129 bytes} [Sat Aug 27 02:09:12 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:13.143517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:13.179075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cfc06b-7e61-419b-a972-42a0514087aa None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:13.240193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:13.300478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a76f5c-60fa-4e65-996d-ff8fc66e31b7 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:13.306451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:13.307591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:13.308099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:13.308597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:13.328981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:13.330688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:13.349060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:13.349811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:13.350316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:13.350545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:13.351218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:13.351506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:13.353233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:13.364902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:13.365949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:13.366845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:13.366845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:13.366845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:13.366845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 385/792] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 404 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:13.368390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 408/793] 10.222.0.52 () {56 vars in 1124 bytes} [Sat Aug 27 02:09:13 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:13.373806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:13.374125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:13.489792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:13.508776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f764dd5-12d0-47a4-8a1a-7907bdb295f3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:13.580247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83add046-4524-4e82-a88d-9f7e1b43fe94 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:13.598088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:13.598713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:13.599004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:13.599268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:13.599907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:13.600206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:13.601609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:13.640932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 386/794] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:13.649385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:13.653903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:13.654537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:13.654813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:13.655083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:13.655737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:13.656123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:13.662403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:13.685717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 409/795] 10.222.0.52 () {56 vars in 1124 bytes} [Sat Aug 27 02:09:13 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:13.690533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:13.720955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:13.878622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eb9eb1-f8db-4c43-ac3d-5fdf3c62222f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:13.929943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1c5b6c-31a6-4ddd-8e36-84af23f35528 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:13.941080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:13.941772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:13.942043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:13.942307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:13.953920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:13.957241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:13.958732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:13.972754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31704857-b51b-4289-aaee-52dc7468dab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:13.977311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31704857-b51b-4289-aaee-52dc7468dab7 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:13.977625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31704857-b51b-4289-aaee-52dc7468dab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:13.977904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31704857-b51b-4289-aaee-52dc7468dab7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:13.978629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31704857-b51b-4289-aaee-52dc7468dab7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:13.982075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:13.982972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:13.983850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 387/796] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:13.984384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31704857-b51b-4289-aaee-52dc7468dab7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:13.989894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:14.038693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 410/797] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:13 2022] POST /identity/v3/endpoints => generated 346 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:14.171320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:14.181985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c977c286-2914-4964-a476-e4745007d6fa None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:14.241966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab56afa-9383-4caf-9435-cc10b5cbd248 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:14.242303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab56afa-9383-4caf-9435-cc10b5cbd248 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:14.242541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab56afa-9383-4caf-9435-cc10b5cbd248 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:14.246444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 411/798] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:14.247391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:14.248258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:14.248746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:14.249262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:14.250614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:14.251879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:14.255747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:14.262224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669f428-db46-4aef-8ccc-25e7e288c178 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:14.262549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669f428-db46-4aef-8ccc-25e7e288c178 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:14.262782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669f428-db46-4aef-8ccc-25e7e288c178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:14.280823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 388/799] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:14.290673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:14.302234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c669f428-db46-4aef-8ccc-25e7e288c178 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:14.303685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c669f428-db46-4aef-8ccc-25e7e288c178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:14.422540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 412/800] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:14.475951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd154ddc-cecf-48b1-b04f-2d5b7bf8e6e0 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:14.476455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd154ddc-cecf-48b1-b04f-2d5b7bf8e6e0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:14.477084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd154ddc-cecf-48b1-b04f-2d5b7bf8e6e0 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:14.480885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfafb8-a3ca-432b-9387-6ba83f902e07 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:14.485290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 413/801] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:14.488307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7640901-9991-495b-8a15-e38f3167bdf0 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:14.488775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7640901-9991-495b-8a15-e38f3167bdf0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:14.489294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7640901-9991-495b-8a15-e38f3167bdf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:14.518104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d7640901-9991-495b-8a15-e38f3167bdf0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:14.519757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d7640901-9991-495b-8a15-e38f3167bdf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:14.562891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e185061d-1508-479a-825f-325891941119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:14.565707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e185061d-1508-479a-825f-325891941119 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:14.566167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e185061d-1508-479a-825f-325891941119 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:14.566626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e185061d-1508-479a-825f-325891941119 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/cfc9a334f972474aa19aa3beae51560f/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:14.570616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e185061d-1508-479a-825f-325891941119 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=cfc9a334f972474aa19aa3beae51560f, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:14.574511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e185061d-1508-479a-825f-325891941119 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:14.608738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 389/802] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:09:14 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/cfc9a334f972474aa19aa3beae51560f/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:09:14.662681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 414/803] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:14.822938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e185061d-1508-479a-825f-325891941119 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:14.843809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:14.969856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e185061d-1508-479a-825f-325891941119 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:15.005722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01654416-7597-4324-88e2-3c37d2e9a391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:15.006360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01654416-7597-4324-88e2-3c37d2e9a391 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.006631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01654416-7597-4324-88e2-3c37d2e9a391 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.006898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01654416-7597-4324-88e2-3c37d2e9a391 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.019624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01654416-7597-4324-88e2-3c37d2e9a391 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-01654416-7597-4324-88e2-3c37d2e9a391 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:15.028097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:15.029758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:15.032618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 390/804] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:09:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:15.046064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7640901-9991-495b-8a15-e38f3167bdf0 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:15.104859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:15.212247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7640901-9991-495b-8a15-e38f3167bdf0 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:15.220465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349908-e629-47af-a2d1-dd5ebec838b8 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.221075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349908-e629-47af-a2d1-dd5ebec838b8 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.221599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349908-e629-47af-a2d1-dd5ebec838b8 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.229585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 391/805] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:15 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:15.235312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4cc2ca-5a85-4806-853a-7df9fddb8070 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.236121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4cc2ca-5a85-4806-853a-7df9fddb8070 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.236604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4cc2ca-5a85-4806-853a-7df9fddb8070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.256502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:15.257113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.257406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.257666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.258336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:15.258682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:15.260056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:15.264041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-cf4cc2ca-5a85-4806-853a-7df9fddb8070 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:15.265986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-cf4cc2ca-5a85-4806-853a-7df9fddb8070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:15.269759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 415/806] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:09:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:15.270982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:15.375987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:15.383884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:15.387499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 392/807] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:15.388665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e0fe44-9027-4362-bc14-cdce2bc28858 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:15.389387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4cc2ca-5a85-4806-853a-7df9fddb8070 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:15.453022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:15.453855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.454330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.454663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.467422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:15.474232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:15.475192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:15.476023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 416/808] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:15.486066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:15.619463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4cc2ca-5a85-4806-853a-7df9fddb8070 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:15.646076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd8df6a-f77b-4630-9e89-1d26dadf671d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:15.690432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:15.691069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.691341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.691613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.692297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:15.692597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:15.694007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:15.709514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:15.713171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.713171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.713171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.713972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 417/809] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:09:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:15.723806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:15.732599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:15.734030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:15.765188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:15.767082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:15.768773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:15.768773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:15.768773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:15.780950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 393/810] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 385 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:15.780950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:15.782031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:15.878502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:15.904081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69461a51-ac65-4d2f-8c97-c37f72e79863 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:15.957625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec721b01-16f2-4615-a4b7-22691c876021 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:15.958755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:15.960257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:15.960894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:15.961554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:15.979396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:15.982172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:15.999463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.000681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.000965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.001268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.001948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:16.002252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.004012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:16.009961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:16.011783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:16.013625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:16.013625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:16.013625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:16.016435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 418/811] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:16.018086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.019587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 394/812] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:16.026254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.120239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:16.159766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e1d6ad-0328-4f71-ad91-42f7c5a02119 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.173693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50e4bbd-d2c7-448a-a794-b9a2570ab024 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.180819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:16.208713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.209712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.210132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.210569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.220434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.221487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.221998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.222289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] PATH_INFO: `/v3/services/network` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.222972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.224186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.224543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.226678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:16.234996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:16.236367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 27 02:09:16.237339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:16.243011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 419/813] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/services/network => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:16.244109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:16.244311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:16.245707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:16.247558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:16.251408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 395/814] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:16.252558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.364941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc37fed8-ff0a-4829-baf3-c25f920c33a1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.383979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898f77f9-1787-4095-a270-e56620803d0b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.402998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.403862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.404313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.404741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.405584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] RBAC: Adding query filter params (name=network) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:16.406323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.407858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.418100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 396/815] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/services?name=network => generated 124 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:16.419366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.422317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.423070 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.423352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.423675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.425001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:16.425001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.426560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.429660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:16.443425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 420/816] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:16.444452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.462074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:16.535264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915e82a-464c-4e6a-8a09-c9dcce548553 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.572525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.573390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.573877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.574197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.575895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:16.576223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.578922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.582427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f7cdc0-bc44-4b29-b1da-b3f48c01e3ae None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.600873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 397/817] 10.222.0.52 () {56 vars in 1149 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:16.606381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.636319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.637756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.638223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.638767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.639790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] RBAC: Adding query filter params (type=network) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:16.640271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.642793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.650004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 421/818] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/services?type=network => generated 350 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:16.651092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.739257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:16.744508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:16.748629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad00d3-b18b-4eda-aea5-f64d14fb7607 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.791122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19581da1-0ac9-426d-b9fe-9b43d3c3c15a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:16.793616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.794952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.794952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.794952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.803565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.805146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:16.819983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:16.821315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 398/819] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:16.822409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.831586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea6d282-7434-4dce-9ea9-cbb7d2806e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:16.832233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea6d282-7434-4dce-9ea9-cbb7d2806e97 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.832505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea6d282-7434-4dce-9ea9-cbb7d2806e97 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.832775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea6d282-7434-4dce-9ea9-cbb7d2806e97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.843906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea6d282-7434-4dce-9ea9-cbb7d2806e97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:16.845504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea6d282-7434-4dce-9ea9-cbb7d2806e97 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-7ea6d282-7434-4dce-9ea9-cbb7d2806e97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:16.866134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:16.867143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:16.868073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:16.868073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:16.868073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:16.868727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 422/820] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:16.872490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2bb0e-4a25-4a6d-a68d-798d81a4d37f None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:16.872956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2bb0e-4a25-4a6d-a68d-798d81a4d37f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:16.875935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2bb0e-4a25-4a6d-a68d-798d81a4d37f None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:16.877447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 423/821] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:16 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:16.878384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d2bb0e-4a25-4a6d-a68d-798d81a4d37f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:16.990023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc01fee4-500b-4180-a634-8020f757597e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:17.023349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d2bb0e-4a25-4a6d-a68d-798d81a4d37f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:17.038244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:17.038859 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.039136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.039405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.040061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:17.040523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:17.041552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:17.075126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 399/822] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:16 2022] POST /identity/v3/endpoints => generated 337 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:17.076597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:17.095315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583b43f5-63ab-41a7-bb81-12e7d144c766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:17.096636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583b43f5-63ab-41a7-bb81-12e7d144c766 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.097382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583b43f5-63ab-41a7-bb81-12e7d144c766 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.097983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583b43f5-63ab-41a7-bb81-12e7d144c766 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.098838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:17.099793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583b43f5-63ab-41a7-bb81-12e7d144c766 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:17.101071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583b43f5-63ab-41a7-bb81-12e7d144c766 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:17.104260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583b43f5-63ab-41a7-bb81-12e7d144c766 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:17.139651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 424/823] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:09:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:17.141294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55611ed6-0c6c-46d8-8d28-44224f112aff None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.141294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55611ed6-0c6c-46d8-8d28-44224f112aff None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.141614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55611ed6-0c6c-46d8-8d28-44224f112aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.177418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-55611ed6-0c6c-46d8-8d28-44224f112aff None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:17.178684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-55611ed6-0c6c-46d8-8d28-44224f112aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:17.247116 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b9b215-f0f8-4968-b97e-3afff7db1adc None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:17.288085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 425/824] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:17 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:17.289548 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55611ed6-0c6c-46d8-8d28-44224f112aff None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:17.305494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfd380c-efeb-4424-b19a-763ec8c2b750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:17.306155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd380c-efeb-4424-b19a-763ec8c2b750 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.306504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd380c-efeb-4424-b19a-763ec8c2b750 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.306999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfd380c-efeb-4424-b19a-763ec8c2b750 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.308071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfd380c-efeb-4424-b19a-763ec8c2b750 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:17.308553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfd380c-efeb-4424-b19a-763ec8c2b750 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:17.310812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfd380c-efeb-4424-b19a-763ec8c2b750 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:17.331992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 400/825] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:17.335910 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb625e0a-1c3e-4a03-b03f-16a9fe779b33 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.336810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb625e0a-1c3e-4a03-b03f-16a9fe779b33 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.338032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb625e0a-1c3e-4a03-b03f-16a9fe779b33 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.341072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 401/826] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:17 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:17.343989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb625e0a-1c3e-4a03-b03f-16a9fe779b33 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:17.460132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:17.471804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:17.476374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55611ed6-0c6c-46d8-8d28-44224f112aff None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:17.506661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb625e0a-1c3e-4a03-b03f-16a9fe779b33 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:17.519076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:17.519507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.519782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.520051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.544937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fab3c9c-a7f8-4022-bcce-0538ef6f9195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:17.545724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab3c9c-a7f8-4022-bcce-0538ef6f9195 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.546182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab3c9c-a7f8-4022-bcce-0538ef6f9195 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.546648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab3c9c-a7f8-4022-bcce-0538ef6f9195 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.559405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fab3c9c-a7f8-4022-bcce-0538ef6f9195 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:17.562052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fab3c9c-a7f8-4022-bcce-0538ef6f9195 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-9fab3c9c-a7f8-4022-bcce-0538ef6f9195 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:17.579049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:17.580068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:17.580938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:17.585305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 402/827] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:17.587989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0b36a-60b5-45ff-8d12-6d24332ff541 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.588790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0b36a-60b5-45ff-8d12-6d24332ff541 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.589291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0b36a-60b5-45ff-8d12-6d24332ff541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.606053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:17.608706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:17.609759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-78e0b36a-60b5-45ff-8d12-6d24332ff541 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:17.611112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-78e0b36a-60b5-45ff-8d12-6d24332ff541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:17.683484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 403/828] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:17 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:17.684333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0b36a-60b5-45ff-8d12-6d24332ff541 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:17.728791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:17.748641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 426/829] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:17 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:09:17.748641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:17.764722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:17.819133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0b36a-60b5-45ff-8d12-6d24332ff541 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:17.876495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:17.877489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.878217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.878924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.879429 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:17.879756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:17.881240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:17.894478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 404/830] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:17.895420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:17.904289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a181b5-f643-41eb-aad8-21a46ea7bdef None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:17.964019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:17.964655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:17.964936 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:17.965264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:17.965828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:17.967224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:17.976679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:17.981816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:17.982904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 427/831] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:17 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:17.983942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.041868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed27c22-ec13-4185-8b31-39c4703d670c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.066678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:18.074062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:18.090543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.091708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.092153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.092586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.111123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:18.113102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:18.129687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:18.131490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:18.133163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:18.133163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:18.133163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:18.133502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81b3446-7e44-4d1d-a6c9-f72108f3000c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.133633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 405/832] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:18.133633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.174646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.175317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.175584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.175849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/1a14c9fa11f94a3ab0c4d32f6f8f0931/roles/439d280788384429bc6d5ff11a303b76` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.192195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=1a14c9fa11f94a3ab0c4d32f6f8f0931, role_id=439d280788384429bc6d5ff11a303b76)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:18.193698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:18.215940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 428/833] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:09:17 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/1a14c9fa11f94a3ab0c4d32f6f8f0931/roles/439d280788384429bc6d5ff11a303b76 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:09:18.217303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.303053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f3664-2c9d-44b2-b70c-0751244f0d46 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.359526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.361026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.361856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.362447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.371679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd46d850-b3c6-4f47-a7ff-6b840b8c6394 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.380538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:18.386646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:18.387830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:18.388839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:18.390216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 406/834] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:18.392442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.396706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:18.427306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.428204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.428491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.428740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.429421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:18.429724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:18.431692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:18.434729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:18.437951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 429/835] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:18.438858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.522990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c65ef3-158c-42a8-9f43-32ce4920f86c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.567364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.568316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.568921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.569401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.570596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:18.571072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:18.572917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:18.584358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec01fc-8136-4058-b64a-5d17e808a6f6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.595448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 407/836] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:18.595448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.638080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.638984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.639430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.640663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.641623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:18.641928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:18.643716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:18.657271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 430/837] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:18.657525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.716368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:18.731824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:18.745701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68e1000-fab5-4291-99b3-d5bf3962c378 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.791331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.792003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.792206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.792480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.802460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:18.802460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:18.802460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:18.817165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 408/838] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:18.817165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.836465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e014a99a-91ee-49c5-be0b-c6b558387775 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:18.892681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:18.893834 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:18.894260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:18.894669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:18.909166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:18.912998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:18.936973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:18.938519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:18.939435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:18.941396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 431/839] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:18.952491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:18.976995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c0fe7-6995-423d-9692-5da87fc9dd78 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:19.041930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:19.042570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:19.042845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:19.043932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:19.059942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:19.066482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:19.086894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:19.087886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:19.088837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:19.088837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:19.088837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:19.088837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 409/840] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:19.097729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:19.104518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:19.111438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7623f1-97cb-4e1f-984f-17d4974fea13 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:19.111649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:19.150224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:19.151588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:19.152161 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:19.152529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:19.153300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:19.153752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:19.155591 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:19.175681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 432/841] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:19.183875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:19.265396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f70e67-78db-4f5c-bbce-4807ba5224ca None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:19.321052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:19.322824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:19.323813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:19.324337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:19.325198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:19.325648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:19.327170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:19.349338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 410/842] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:19.352216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8ed9a7-3a95-4f69-a7ff-23024f880657 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:19.358248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:19.414953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:19.415802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:19.416119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:19.416392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:19.418182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=c83f0d0e2e8047a98fbe114ceb15c6e0) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:19.418672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:19.421403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:19.442719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 433/843] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:19 2022] GET /identity/v3/role_assignments?user.id=c83f0d0e2e8047a98fbe114ceb15c6e0&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:19.479819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:19.529246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88791f2-cf2f-48b3-adb4-e186cd85a374 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:19.589660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:19.590455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:19.590750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:19.591011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] PATH_INFO: `/v3/users/swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:19.602458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:19.603955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:19.624239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:19.626183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:19.628063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 411/844] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:19.634134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:19.685060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:19.786538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fbbe52-ec3d-42de-ab36-69dd86020f76 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:19.835845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:19.836682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:19.837056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:19.837456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:19.838498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:19.838916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:19.840993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:19.872843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 434/845] 10.222.0.52 () {56 vars in 1135 bytes} [Sat Aug 27 02:09:19 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:19.882898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:20.046611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb69bb4-06da-4eef-8c80-03b88c9678de None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:20.091336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:20.111504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:20.112126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:20.112397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:20.112663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:20.213354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:20.214963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:20.326073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 412/846] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:19 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:09:20.332180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:20.421723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:20.449200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314cacfc-7273-466d-b1b4-bc1538fca3e9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:20.490668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2271d2a-cfc9-41c8-a8b3-a671b377e886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:20.491317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2271d2a-cfc9-41c8-a8b3-a671b377e886 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:20.491593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2271d2a-cfc9-41c8-a8b3-a671b377e886 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:20.491937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2271d2a-cfc9-41c8-a8b3-a671b377e886 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:20.506603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2271d2a-cfc9-41c8-a8b3-a671b377e886 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:20.508208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2271d2a-cfc9-41c8-a8b3-a671b377e886 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c2271d2a-cfc9-41c8-a8b3-a671b377e886 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:20.527798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:20.529013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:20.530314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:20.530314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:20.530314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:20.530314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 435/847] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:20.538493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:20.578784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:20.691637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a881417-469a-46aa-a96b-1741373798d3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:20.726777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a280c9-25e8-4904-9b51-b080b0af2192 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:20.727367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a280c9-25e8-4904-9b51-b080b0af2192 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:20.727367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a280c9-25e8-4904-9b51-b080b0af2192 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:20.729632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 436/848] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:20.736276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f6b2b-53d5-4895-8ae9-8e2433e19f37 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:20.736678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f6b2b-53d5-4895-8ae9-8e2433e19f37 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:20.736934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2f6b2b-53d5-4895-8ae9-8e2433e19f37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:20.748149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:20.749255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:20.749730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:20.750296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:20.751896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:20.752434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:20.755070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:20.760377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-aa2f6b2b-53d5-4895-8ae9-8e2433e19f37 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:20.762477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-aa2f6b2b-53d5-4895-8ae9-8e2433e19f37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:20.769987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 413/849] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:20.778743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:20.909709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 437/850] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:20 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:20.911596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:20.927325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08335ec8-d13d-4a1a-9a32-7f3ed42be062 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:20.973482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:20.974498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:20.974999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:20.975497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:20.983918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:20.985886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:21.000560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:21.002000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:21.003822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:21.006623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 414/851] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:21.017638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2f6b2b-53d5-4895-8ae9-8e2433e19f37 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:21.107806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:21.155510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2f6b2b-53d5-4895-8ae9-8e2433e19f37 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:21.247423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:21.386641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97edb938-0e7b-4888-bc22-1472339b3228 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:21.387075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:21.388081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:21.388684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:21.389191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:21.390739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:21.391269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:21.393966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:21.418847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 438/852] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 405 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:21.426705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:21.427466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:21.428304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:21.428586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:21.429157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:21.449667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:21.455594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:21.456634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:09:21.457720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:21.462330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 415/853] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:09:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:21.463286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:21.478194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:21.594141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc70dcf-1d0f-4bbb-a6f9-d36c37e0d506 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:21.619191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ae4c3-6dc7-4a4b-928e-e0a0bf27cacc None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:21.620587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:21.646303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8bde69-3d38-47bf-8b4d-86e47fff8100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:21.647146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8bde69-3d38-47bf-8b4d-86e47fff8100 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:21.647439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8bde69-3d38-47bf-8b4d-86e47fff8100 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:21.647726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8bde69-3d38-47bf-8b4d-86e47fff8100 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:21.648456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8bde69-3d38-47bf-8b4d-86e47fff8100 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=cfc9a334f972474aa19aa3beae51560f) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:21.648775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8bde69-3d38-47bf-8b4d-86e47fff8100 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:21.650990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8bde69-3d38-47bf-8b4d-86e47fff8100 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:21.663814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 439/854] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:21 2022] GET /identity/v3/role_assignments?user.id=cfc9a334f972474aa19aa3beae51560f&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:21.663814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55995ae-8ba1-4227-b03c-3daf2600bccc None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:21.663814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55995ae-8ba1-4227-b03c-3daf2600bccc None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:21.663814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55995ae-8ba1-4227-b03c-3daf2600bccc None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:21.663814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 440/855] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:21.674505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b3c34-9fc6-4750-a451-093b6e631846 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:21.674817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b3c34-9fc6-4750-a451-093b6e631846 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:21.675119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b3c34-9fc6-4750-a451-093b6e631846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:21.683456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:21.685027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:21.686980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:21.687733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:21.689238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:21.689880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:21.692221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:21.699482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-212b3c34-9fc6-4750-a451-093b6e631846 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:21.700854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-212b3c34-9fc6-4750-a451-093b6e631846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:21.710248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 416/856] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:09:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:21.724466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:21.836407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:21.857376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 441/857] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:21 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:21.894391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a9de99-e813-4aa7-9bdf-6e3929d725e4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:21.948858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:21.950599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:21.951081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:21.951695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:21.969436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:21.971881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:21.991109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:21.992151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:21.993281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:21.993281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:21.993281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:21.996159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 417/858] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:22.005435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212b3c34-9fc6-4750-a451-093b6e631846 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:22.138397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212b3c34-9fc6-4750-a451-093b6e631846 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:22.160769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:22.184731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:22.185392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:22.185669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:22.185939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:22.186605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:22.186904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:22.188378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:22.201974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 442/859] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:22.208238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:22.274927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:22.401694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115232e6-5909-44cf-8979-6060af3236d3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:22.459160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e633f4c9-ecaf-49bd-a6f4-5b4aa4e41d1a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:22.459451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:22.481789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2921f64b-2a3e-4f09-888e-4b259c4306fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:22.482701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921f64b-2a3e-4f09-888e-4b259c4306fb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:22.483030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921f64b-2a3e-4f09-888e-4b259c4306fb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:22.483224 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921f64b-2a3e-4f09-888e-4b259c4306fb None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:22.492635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2921f64b-2a3e-4f09-888e-4b259c4306fb None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:22.494384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2921f64b-2a3e-4f09-888e-4b259c4306fb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:22.505745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef9e47c-145f-4528-b501-4dd8ded8067d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:22.506351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef9e47c-145f-4528-b501-4dd8ded8067d None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:22.506638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef9e47c-145f-4528-b501-4dd8ded8067d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:22.506885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef9e47c-145f-4528-b501-4dd8ded8067d None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:22.507521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef9e47c-145f-4528-b501-4dd8ded8067d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-2921f64b-2a3e-4f09-888e-4b259c4306fb None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:22.508306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:22.510291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:22.512115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:22.513279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef9e47c-145f-4528-b501-4dd8ded8067d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:22.516903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 418/860] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:09:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:22.518754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9629a4c5-4c72-49f6-880d-b854358ee8a4 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:22.519270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9629a4c5-4c72-49f6-880d-b854358ee8a4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:22.519685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9629a4c5-4c72-49f6-880d-b854358ee8a4 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:22.526956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 419/861] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:22.526956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9629a4c5-4c72-49f6-880d-b854358ee8a4 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:22.536947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 443/862] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:22 2022] POST /identity/v3/services => generated 251 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:09:22.541372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea46b414-744e-40cb-82df-888b11aa0e23 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:22.541372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea46b414-744e-40cb-82df-888b11aa0e23 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:22.541372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea46b414-744e-40cb-82df-888b11aa0e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:22.568727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ea46b414-744e-40cb-82df-888b11aa0e23 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:22.570584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ea46b414-744e-40cb-82df-888b11aa0e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:22.615025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:22.724016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 444/863] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:22 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:22.736685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9629a4c5-4c72-49f6-880d-b854358ee8a4 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:22.807202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:22.808257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:22.808746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:22.809403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:22.810817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:22.811382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:22.814115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:22.856502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 420/864] 10.222.0.52 () {56 vars in 1149 bytes} [Sat Aug 27 02:09:22 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:22.860741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea46b414-744e-40cb-82df-888b11aa0e23 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:23.023953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea46b414-744e-40cb-82df-888b11aa0e23 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:23.086073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:23.086977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:23.087379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:23.087777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:23.094087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:23.137565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:23.178481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:23.190798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:23.192722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:23.270066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804e731-ea54-46a9-a399-ebe099dd3a81 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:23.306586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:23.307138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:23.307413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:23.307675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:23.308261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:23.309670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:23.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:23.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:23.321890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 421/865] 10.222.0.52 () {56 vars in 1132 bytes} [Sat Aug 27 02:09:23 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:23.332329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:23.355688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 445/866] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:22 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 496 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:09:23.363765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:23.486324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:23.510721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30404d9-a631-4f73-aa86-bc8d956b54f0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:23.516032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:23.575641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7261c-79dd-4696-98e2-468e56b0dfdd None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:23.581688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:23.582724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:23.583188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:23.583637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:23.584796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:23.585443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:23.587483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:23.601535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 422/867] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:23 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:23.610706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:23.666924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:23.668105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:23.668511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:23.668926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:23.686972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:23.689520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:23.720102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:23.721617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:23.723041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:23.723041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:23.723041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:23.726050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 446/868] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 360 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:23.727395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:23.777725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bbbbd8-7e40-4029-bd94-0762aa94a9d1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:23.829737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:23.830905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:23.831475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:23.831947 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:23.832807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:23.833370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:23.835206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:23.845543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 423/869] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:23 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:23.851933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e4d5c-f657-4f13-841d-b241bd0da984 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:23.876390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:23.913271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3683f60b-486e-4eee-9a27-adeffcb25f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:23.914166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683f60b-486e-4eee-9a27-adeffcb25f6e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:23.914715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683f60b-486e-4eee-9a27-adeffcb25f6e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:23.915158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683f60b-486e-4eee-9a27-adeffcb25f6e None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:23.916270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3683f60b-486e-4eee-9a27-adeffcb25f6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:23.916721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3683f60b-486e-4eee-9a27-adeffcb25f6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:23.918715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3683f60b-486e-4eee-9a27-adeffcb25f6e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:23.945331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 447/870] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:23.950300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:24.005502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:24.129903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf90c68-d80a-46b5-9a87-005eb036c8d6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:24.192908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:24.194001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:24.194476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:24.194940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:24.207800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:24.210475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:24.215139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a074e1-c501-402c-b882-2fddea54c485 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:24.215499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a074e1-c501-402c-b882-2fddea54c485 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:24.215749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a074e1-c501-402c-b882-2fddea54c485 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:24.217803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 448/871] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:24.226236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70dfb1-f1f8-44a3-b266-42ef7fd669d8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:24.226568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70dfb1-f1f8-44a3-b266-42ef7fd669d8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:24.226858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70dfb1-f1f8-44a3-b266-42ef7fd669d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:24.233112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:24.234058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:24.235010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:24.235726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 424/872] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:24.243881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:24.263322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ea70dfb1-f1f8-44a3-b266-42ef7fd669d8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:24.265595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ea70dfb1-f1f8-44a3-b266-42ef7fd669d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:24.394808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 449/873] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:24 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:24.463937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b1014e-3bf5-4277-a0b8-270b96bbc998 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:24.466793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:24.525406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:24.526344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:24.526800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:24.527263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:24.528496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:24.529010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:24.532140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:24.560595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 425/874] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:24.565078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70dfb1-f1f8-44a3-b266-42ef7fd669d8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:24.761154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70dfb1-f1f8-44a3-b266-42ef7fd669d8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:24.763589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:24.816864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:24.817779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:24.818188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:24.818593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:24.819729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:24.820194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:24.822650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:24.837372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 450/875] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:24 2022] GET /identity/v3/role_assignments?user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 608 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:24.849715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:25.023792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8cdfd9-eaaa-4909-bfdc-cebb7ff81eaf None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:25.072571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f6bddf-cddb-471a-9b5a-1012c0943516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:25.073605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f6bddf-cddb-471a-9b5a-1012c0943516 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:25.074028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f6bddf-cddb-471a-9b5a-1012c0943516 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:25.074432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f6bddf-cddb-471a-9b5a-1012c0943516 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:25.075343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f6bddf-cddb-471a-9b5a-1012c0943516 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:25.077539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f6bddf-cddb-471a-9b5a-1012c0943516 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-95f6bddf-cddb-471a-9b5a-1012c0943516 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:25.092504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:25.093571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:25.094447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:25.095297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 426/876] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:24 2022] GET /identity/v3/services/object-store => generated 93 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:25.101473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:25.135843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb3621-4647-42f1-81f3-f44830914b7f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:25.136425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb3621-4647-42f1-81f3-f44830914b7f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:25.136425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb3621-4647-42f1-81f3-f44830914b7f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:25.138578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 427/877] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:25.145784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec50e4e-4fe7-49eb-9670-0b3f140dabcc None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:25.146101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec50e4e-4fe7-49eb-9670-0b3f140dabcc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:25.146413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec50e4e-4fe7-49eb-9670-0b3f140dabcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:25.166524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7ec50e4e-4fe7-49eb-9670-0b3f140dabcc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:25.167822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7ec50e4e-4fe7-49eb-9670-0b3f140dabcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:25.186443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:25.280740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 428/878] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:25 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:25.295628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b7c4f-4149-4e2c-b1dd-a2a0489162b7 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:25.335683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:25.336311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:25.336626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:25.336876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:25.337539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:25.338038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:25.339848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:25.347545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 451/879] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:25 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:25.353639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec50e4e-4fe7-49eb-9670-0b3f140dabcc None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:25.489041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec50e4e-4fe7-49eb-9670-0b3f140dabcc None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:25.532932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:25.533489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:25.533757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:25.534030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:25.534801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:25.535115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:25.536486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:25.542811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 429/880] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:25 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:25.671208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:25.707263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:25.790113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa840ee7-5237-4ea7-b831-31dd5c6665ac None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:25.826426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:25.827455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:25.827927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:25.828483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:25.829439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:25.831004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:25.840723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:25.841780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:25.842602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:25.845476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 452/881] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:25 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:25.851351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:25.865308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:25.990502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aa983e-c11f-4797-b7bd-a6a21aabac78 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:26.034750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:26.035760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:26.036235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:26.036735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:26.037669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:26.038185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:26.039803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:26.046272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 430/882] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:25 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:26.051899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:26.178588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:26.187349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f570db-7736-40cb-93e3-6127d39f6005 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:26.235658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:26.236700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:26.237256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:26.237748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:26.238752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:26.239369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:26.241096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:26.247762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 453/883] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:26 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:26.255707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:26.347984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:26.379264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cf13df-a974-46a8-90f8-527e3162a664 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:26.417067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55fe89-3c18-4e1d-aced-9cf83f936dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:26.417741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55fe89-3c18-4e1d-aced-9cf83f936dde None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:26.418028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55fe89-3c18-4e1d-aced-9cf83f936dde None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:26.418304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55fe89-3c18-4e1d-aced-9cf83f936dde None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:26.419041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d55fe89-3c18-4e1d-aced-9cf83f936dde None admin] RBAC: Adding query filter params (interface=public, service_id=f40c22a2ceeb4348a6c2233250a8b0fb, region_id=RegionOne) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:26.419349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d55fe89-3c18-4e1d-aced-9cf83f936dde None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:26.420727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d55fe89-3c18-4e1d-aced-9cf83f936dde None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:26.427759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 431/884] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:09:26 2022] GET /identity/v3/endpoints?service_id=f40c22a2ceeb4348a6c2233250a8b0fb&interface=public®ion_id=RegionOne => generated 194 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:26.435750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:26.595814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdd8f3-003f-44ec-9c20-d6f8a2d1e69f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:26.649347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fa1511-c1d4-4552-ba33-6f4c15d92055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:26.650205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa1511-c1d4-4552-ba33-6f4c15d92055 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:26.650677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa1511-c1d4-4552-ba33-6f4c15d92055 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:26.651141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa1511-c1d4-4552-ba33-6f4c15d92055 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:26.652083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fa1511-c1d4-4552-ba33-6f4c15d92055 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:26.654462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fa1511-c1d4-4552-ba33-6f4c15d92055 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:26.663924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 454/885] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:26 2022] GET /identity/v3/services => generated 1011 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:09:26.805454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba49073-d587-49ed-9dae-b0e5462b3bbb None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:26.805981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba49073-d587-49ed-9dae-b0e5462b3bbb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:26.806473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba49073-d587-49ed-9dae-b0e5462b3bbb None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:26.809693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 432/886] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:26.819422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddd9592-94ee-4c52-a387-91a10414c82e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:26.819915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddd9592-94ee-4c52-a387-91a10414c82e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:26.820404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddd9592-94ee-4c52-a387-91a10414c82e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:26.852029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4ddd9592-94ee-4c52-a387-91a10414c82e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:26.854087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4ddd9592-94ee-4c52-a387-91a10414c82e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:26.877757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:26.968596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 455/887] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:26 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:27.381077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba49073-d587-49ed-9dae-b0e5462b3bbb None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:27.561873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba49073-d587-49ed-9dae-b0e5462b3bbb None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:27.594798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d9ffa3-87b7-4ee1-a201-533132f87f77 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:27.595490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d9ffa3-87b7-4ee1-a201-533132f87f77 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:27.595743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d9ffa3-87b7-4ee1-a201-533132f87f77 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:27.605075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 456/888] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:27 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:27.607340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9e64f-4a85-4ee4-b3d2-db6a931d904e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:27.607829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9e64f-4a85-4ee4-b3d2-db6a931d904e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:27.608316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9e64f-4a85-4ee4-b3d2-db6a931d904e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:27.630479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a79ca75-1fbb-48bc-af84-bb49080a10f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:27.631903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79ca75-1fbb-48bc-af84-bb49080a10f6 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:27.632403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79ca75-1fbb-48bc-af84-bb49080a10f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:27.632973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a79ca75-1fbb-48bc-af84-bb49080a10f6 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:27.634645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a79ca75-1fbb-48bc-af84-bb49080a10f6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:27.637958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a79ca75-1fbb-48bc-af84-bb49080a10f6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:27.648197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3fb9e64f-4a85-4ee4-b3d2-db6a931d904e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:27.650324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3fb9e64f-4a85-4ee4-b3d2-db6a931d904e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:27.651300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:27.665156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 433/889] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:27 2022] POST /identity/v3/services => generated 269 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:27.805353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 457/890] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:27 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:28.136950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba98677-f98a-4a05-8230-e4889c21e005 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:28.136950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba98677-f98a-4a05-8230-e4889c21e005 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:28.136950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba98677-f98a-4a05-8230-e4889c21e005 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:28.137926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 434/891] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:28.148399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e533255-c132-4542-95c4-3889e5a6a917 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:28.149417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e533255-c132-4542-95c4-3889e5a6a917 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:28.150197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e533255-c132-4542-95c4-3889e5a6a917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:28.171766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-0e533255-c132-4542-95c4-3889e5a6a917 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:28.174064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0e533255-c132-4542-95c4-3889e5a6a917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:28.181998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba98677-f98a-4a05-8230-e4889c21e005 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:28.289319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 458/892] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:28 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:28.427461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba98677-f98a-4a05-8230-e4889c21e005 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:28.483293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:28.484604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:28.485286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:28.485886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:28.504804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:28.513489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:28.515323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:28.517418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:28.520992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 435/893] 10.222.0.52 () {56 vars in 1120 bytes} [Sat Aug 27 02:09:28 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:28.529194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e533255-c132-4542-95c4-3889e5a6a917 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:28.545575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:28.610386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:28.705111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e533255-c132-4542-95c4-3889e5a6a917 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:28.815213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:28.820859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:28.823475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:28.826381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:28.827364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:28.830339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:28.832645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:28.835857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:28.873482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 459/894] 10.222.0.52 () {56 vars in 1115 bytes} [Sat Aug 27 02:09:28 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:28.874848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:28.907312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce336d-06b6-4327-b0e8-0062cdd0bb1f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:28.984746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:28.985750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:28.986241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:28.986835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:28.988092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:28.990710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:29.023674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 436/895] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:28 2022] POST /identity/v3/services => generated 241 bytes in 411 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:29.059090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:29.109186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62072bb9-cae9-4a14-bca6-5be34ebec7bf None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:29.180348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf7b5a0-3979-4264-ad20-b1dbd104abaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:29.181835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7b5a0-3979-4264-ad20-b1dbd104abaf None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:29.182510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7b5a0-3979-4264-ad20-b1dbd104abaf None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:29.183163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7b5a0-3979-4264-ad20-b1dbd104abaf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:29.202508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf7b5a0-3979-4264-ad20-b1dbd104abaf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:29.205091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf7b5a0-3979-4264-ad20-b1dbd104abaf None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-1cf7b5a0-3979-4264-ad20-b1dbd104abaf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:29.221967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:29.223028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:29.223941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:29.223941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:29.223941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:29.226342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 460/896] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:29.234439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:29.258221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61ddce7-4771-45ac-b5a5-69332ed61489 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:29.258523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61ddce7-4771-45ac-b5a5-69332ed61489 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:29.258793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61ddce7-4771-45ac-b5a5-69332ed61489 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:29.261428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 461/897] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:29.267842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9048d17e-2274-460d-967e-3a49b802e971 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:29.268260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9048d17e-2274-460d-967e-3a49b802e971 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:29.268670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9048d17e-2274-460d-967e-3a49b802e971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:29.292955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-9048d17e-2274-460d-967e-3a49b802e971 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:29.294323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9048d17e-2274-460d-967e-3a49b802e971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:29.387336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 462/898] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:29 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:29.407857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8fd2a4-5d52-4cdf-973e-706b09340f3a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:29.434092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:29.459023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:29.459903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:29.460328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:29.460769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:29.461610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:29.462054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:29.463562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:29.483383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 437/899] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:29.492805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9048d17e-2274-460d-967e-3a49b802e971 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:29.564637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:29.632808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9048d17e-2274-460d-967e-3a49b802e971 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:29.670940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:29.671589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:29.671860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:29.672428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:29.681026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:29.682519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:29.695863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:29.698220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:29.700687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 463/900] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:09:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:29.709308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:29.728177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:29.888600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8737353-6ef6-45c4-94c2-c443b45a278c None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:29.915954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bbddcc-a4f1-4638-9d29-7d004292db34 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:29.937738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:29.964281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:29.965628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:29.966431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:29.966946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:29.969052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:29.969514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:29.972709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:29.987179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:29.988561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:29.989470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:29.990089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:29.991360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:29.994079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:30.010775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 438/901] 10.222.0.52 () {56 vars in 1149 bytes} [Sat Aug 27 02:09:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:30.011738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:30.013419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:30.015301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 27 02:09:30.017023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:30.023318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 464/902] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:29 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:30.030407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:30.063461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:30.157969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a519c-e2c6-46f1-af54-861f2b800c6f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:30.218946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f4837-d9dd-4624-8838-77ea7e4c8ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:30.220032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f4837-d9dd-4624-8838-77ea7e4c8ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:30.220468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f4837-d9dd-4624-8838-77ea7e4c8ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:30.220901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f4837-d9dd-4624-8838-77ea7e4c8ee0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:30.242209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d962b8-f837-4cd2-98dc-72dd0d0ebd3f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:30.300033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1688e8-27a2-48eb-86ba-78df43e2ee3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:30.300959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1688e8-27a2-48eb-86ba-78df43e2ee3a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:30.301422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1688e8-27a2-48eb-86ba-78df43e2ee3a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:30.301847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1688e8-27a2-48eb-86ba-78df43e2ee3a None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:30.302877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1688e8-27a2-48eb-86ba-78df43e2ee3a None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:30.303361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1688e8-27a2-48eb-86ba-78df43e2ee3a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:30.305547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1688e8-27a2-48eb-86ba-78df43e2ee3a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:30.314060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 465/903] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:30 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:30.315575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d430fbf-0307-4c88-988d-783dc7e737f2 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:30.316352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d430fbf-0307-4c88-988d-783dc7e737f2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:30.316804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d430fbf-0307-4c88-988d-783dc7e737f2 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:30.324102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 466/904] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:30.324102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d430fbf-0307-4c88-988d-783dc7e737f2 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:30.339286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031f4837-d9dd-4624-8838-77ea7e4c8ee0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:30.341085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031f4837-d9dd-4624-8838-77ea7e4c8ee0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:30.379502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:30.388606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:30.469710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 439/905] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:30 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:09:30.470228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d430fbf-0307-4c88-988d-783dc7e737f2 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:30.470332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0b07ee-a06f-4c00-84a1-b7cff840b4f4 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:30.470332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0b07ee-a06f-4c00-84a1-b7cff840b4f4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:30.470332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0b07ee-a06f-4c00-84a1-b7cff840b4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:30.485943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8a0b07ee-a06f-4c00-84a1-b7cff840b4f4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:30.487376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8a0b07ee-a06f-4c00-84a1-b7cff840b4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:30.509486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:30.510196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:30.510532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:30.510860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:30.511558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:30.511933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:30.515136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:30.520102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 467/906] 10.222.0.52 () {56 vars in 1114 bytes} [Sat Aug 27 02:09:30 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:30.520521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:30.570312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 440/907] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:30 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:30.571477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0b07ee-a06f-4c00-84a1-b7cff840b4f4 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:30.644976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:30.649971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0efdf6e-defd-4e79-b702-33a663e15997 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:30.680526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:30.698367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:30.699700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:30.700156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:30.700593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:30.720980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:30.724337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:30.751056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0b07ee-a06f-4c00-84a1-b7cff840b4f4 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:30.752212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:30.753676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:30.762305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:30.762305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:30.762305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:30.762305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 468/908] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:30.762305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:30.803879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:30.804680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:30.804950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:30.805371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:30.806082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:30.807459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:30.849976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 441/909] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:30 2022] POST /identity/v3/endpoints => generated 346 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:30.893575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:30.956938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf7574e-5307-473e-aaa9-d807728c1573 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.028869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.029829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.030368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.030981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.032061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:31.032592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.034934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.059786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 469/910] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:31.071013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:31.085457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:31.088371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcffc1b-abd7-465c-aeb1-0149ee52060a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.135767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:31.162185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.163671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.164327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.165002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.166233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.169316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:31.188978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:31.191433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:31.193540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:31.201220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 442/911] 10.222.0.52 () {56 vars in 1132 bytes} [Sat Aug 27 02:09:30 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:31.206624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:31.288061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddba8f6-745c-402f-8cba-8d1177ab5d9b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.368913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.369939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.370488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.371021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.384188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f863d971-357e-4a72-83bb-9d886effd047 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.384803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.387463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:31.410147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:31.412199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:31.414258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:31.421442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 470/912] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:31.425499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:31.428878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.429535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.429812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.430080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.430733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:31.431034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.432417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.437695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 443/913] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:31 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:31.445486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:31.528278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:31.562555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:31.594276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4824d66-2d2e-4687-b541-645a29413816 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.613587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e346a799-3b6e-4138-b4a7-2936d6cfcbad None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.652810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.653937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.654739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.655721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.657445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:31.658265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.661295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.669159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 444/914] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:31 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:31.676252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:31.678223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.679075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.679498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.679891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.680737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:31.681233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.682876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.696896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 471/915] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:31.706274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:31.833895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ef4222-e041-45b8-9944-1c1d2a690832 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.847751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78abe567-2a87-4c01-9d36-deb83b647a55 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:31.869847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:31.882203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:31.894801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb04562-b086-4420-a281-5bdab75b2cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.895712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb04562-b086-4420-a281-5bdab75b2cab None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.896211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb04562-b086-4420-a281-5bdab75b2cab None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.896550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb04562-b086-4420-a281-5bdab75b2cab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.897537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb04562-b086-4420-a281-5bdab75b2cab None admin] RBAC: Adding query filter params (role.id=58422e7671624a43a2f7a18f07bb7506, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:31.898001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb04562-b086-4420-a281-5bdab75b2cab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.898652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:31.899342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.899684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.899844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb04562-b086-4420-a281-5bdab75b2cab None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.900070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.900840 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] RBAC: Adding query filter params (interface=public, service_id=5bd202f1c7ba4c9389074957d72992c1, region_id=RegionOne) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:31.901247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:31.903179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:31.908033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 472/916] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:31 2022] GET /identity/v3/role_assignments?user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=58422e7671624a43a2f7a18f07bb7506 => generated 259 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:31.909551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a403ac-10ba-40ff-a9d4-9b5ea0400a09 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.909961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a403ac-10ba-40ff-a9d4-9b5ea0400a09 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.910333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a403ac-10ba-40ff-a9d4-9b5ea0400a09 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.912095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 473/917] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:31.918728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210158fc-68db-44b4-98de-2eab2dc71fb6 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:31.918977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210158fc-68db-44b4-98de-2eab2dc71fb6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:31.919240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210158fc-68db-44b4-98de-2eab2dc71fb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:31.922528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 445/918] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:09:31 2022] GET /identity/v3/endpoints?service_id=5bd202f1c7ba4c9389074957d72992c1&interface=public®ion_id=RegionOne => generated 194 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:31.931926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:31.953841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-210158fc-68db-44b4-98de-2eab2dc71fb6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:31.957374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-210158fc-68db-44b4-98de-2eab2dc71fb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:32.093659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 474/919] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:32.109264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ec7a2-1416-47e4-938b-181b4dbf675e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:32.165524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:32.166617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:32.167205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:32.167758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:32.168693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:32.170777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:32.180098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 446/920] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:31 2022] GET /identity/v3/services => generated 1497 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:09:32.432412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210158fc-68db-44b4-98de-2eab2dc71fb6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:32.559731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210158fc-68db-44b4-98de-2eab2dc71fb6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:32.599947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:32.618625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:32.619670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:32.620141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:32.620599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:32.621659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:32.624123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:32.641290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:32.642464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:32.643310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:32.644721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 475/921] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:32 2022] GET /identity/v3/services/object-store => generated 93 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:32.651108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:32.710342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:32.787963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438781da-5bc4-47f8-a975-09d470756f44 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:32.830287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:32.831357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:32.831863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:32.832295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:32.833495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:32.834019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:32.836554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:32.845952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 447/922] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:32.854393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:32.971953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2010b0-6d62-4c0d-947f-ec7809e18b72 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:33.009465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c779d99d-f7ec-45dc-9706-ee8ec1bb3916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:33.010076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c779d99d-f7ec-45dc-9706-ee8ec1bb3916 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:33.010353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c779d99d-f7ec-45dc-9706-ee8ec1bb3916 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:33.010621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c779d99d-f7ec-45dc-9706-ee8ec1bb3916 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:33.011261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c779d99d-f7ec-45dc-9706-ee8ec1bb3916 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:33.011559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c779d99d-f7ec-45dc-9706-ee8ec1bb3916 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:33.012923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c779d99d-f7ec-45dc-9706-ee8ec1bb3916 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:33.023197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 476/923] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:33.025577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:33.205003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927e6e79-43f8-4c03-a167-dc89daedd911 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:33.218165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:33.229857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23178c7a-cbae-4a84-b7fe-71a3a22c0398 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:33.230152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23178c7a-cbae-4a84-b7fe-71a3a22c0398 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:33.230336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23178c7a-cbae-4a84-b7fe-71a3a22c0398 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:33.231948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 477/924] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:33.237345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cce8ab3-a890-4c9b-8785-2d9d9f41bd8c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:33.237624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cce8ab3-a890-4c9b-8785-2d9d9f41bd8c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:33.237890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cce8ab3-a890-4c9b-8785-2d9d9f41bd8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:33.256680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-9cce8ab3-a890-4c9b-8785-2d9d9f41bd8c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:33.257911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9cce8ab3-a890-4c9b-8785-2d9d9f41bd8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:33.258574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:33.259190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:33.259465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:33.259733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:33.261249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] RBAC: Adding query filter params (interface=public, service_id=124ee2ed9e84424b8220a84c5ab1aba6, region_id=RegionOne) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:33.261589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:33.262954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:33.269270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 448/925] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:09:33 2022] GET /identity/v3/endpoints?service_id=124ee2ed9e84424b8220a84c5ab1aba6&interface=public®ion_id=RegionOne => generated 194 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:33.275201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:33.317789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:33.346789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 478/926] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:33 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:33.392651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ca60b8-191d-4956-9b60-6b68de11c79a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:33.428139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:33.428755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:33.429030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:33.429329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:33.429954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:33.431319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:33.445279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 449/927] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:33 2022] GET /identity/v3/services => generated 1497 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:09:33.624943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cce8ab3-a890-4c9b-8785-2d9d9f41bd8c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:33.792736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cce8ab3-a890-4c9b-8785-2d9d9f41bd8c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:33.850663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:33.851595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:33.852035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:33.852462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:33.853526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:33.855933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:33.860194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING keystone.server.flask.common [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 27 02:09:33.888477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.sql.core [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 27 02:09:33.888477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 27 02:09:33.888477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [parameters: {'id': 'e02ab64a2e7e4643b34fc9da87fe1dd9', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 27 02:09:33.888477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81745) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e02ab64a2e7e4643b34fc9da87fe1dd9.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e02ab64a2e7e4643b34fc9da87fe1dd9. Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result = self._query(query) Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application conn.query(q) Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 27 02:09:33.903324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result.read() Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 27 02:09:33.904833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application next(self.gen) Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application yield resource Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application next(self.gen) Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.session.rollback() Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.force_reraise() Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise self.value Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application session.commit() Aug 27 02:09:33.906880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._prepare_impl() Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.session.flush() Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._flush(objects) Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application compat.raise_( Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application flush_context.execute() Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 27 02:09:33.908672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rec.execute(self) Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 27 02:09:33.912194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result = self._query(query) Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application conn.query(q) Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application result.read() Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 27 02:09:33.914773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [parameters: {'id': 'e02ab64a2e7e4643b34fc9da87fe1dd9', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:33.924267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:33.926856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e02ab64a2e7e4643b34fc9da87fe1dd9. Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:33.929767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 479/928] 10.222.0.52 () {60 vars in 1140 bytes} [Sat Aug 27 02:09:33 2022] POST /identity/v3/projects => generated 262 bytes in 282 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:33.931483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:33.960519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:34.062505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef8c2f-02a6-48d0-b7b2-f67e2636245b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:34.109339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c01c78d-5f4c-4700-961a-f355fb5cbc49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:34.109988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01c78d-5f4c-4700-961a-f355fb5cbc49 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:34.110472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01c78d-5f4c-4700-961a-f355fb5cbc49 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:34.110948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01c78d-5f4c-4700-961a-f355fb5cbc49 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:34.121723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c01c78d-5f4c-4700-961a-f355fb5cbc49 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:34.124348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c01c78d-5f4c-4700-961a-f355fb5cbc49 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-0c01c78d-5f4c-4700-961a-f355fb5cbc49 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:34.144070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:34.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:34.150906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 450/929] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:34.152731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:34.196819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:34.273638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3650cc-6d07-4edc-a89c-d08d79a7abeb None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:34.313328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5e938-58f6-463b-83fb-ccc67d20e410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:34.314028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5e938-58f6-463b-83fb-ccc67d20e410 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:34.314307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5e938-58f6-463b-83fb-ccc67d20e410 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:34.314565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5e938-58f6-463b-83fb-ccc67d20e410 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:34.315235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f5e938-58f6-463b-83fb-ccc67d20e410 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:34.315531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f5e938-58f6-463b-83fb-ccc67d20e410 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:34.317540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f5e938-58f6-463b-83fb-ccc67d20e410 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:34.347111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 480/930] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:34 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:34.725355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da886b24-59be-40af-8939-acfcd1a6da97 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:34.725708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da886b24-59be-40af-8939-acfcd1a6da97 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:34.726148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da886b24-59be-40af-8939-acfcd1a6da97 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:34.728874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 451/931] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:34.736155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24acf964-712a-481e-9a92-b97ee605dbab None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:34.736625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24acf964-712a-481e-9a92-b97ee605dbab None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:34.737050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24acf964-712a-481e-9a92-b97ee605dbab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:34.761545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-24acf964-712a-481e-9a92-b97ee605dbab None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:34.763090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-24acf964-712a-481e-9a92-b97ee605dbab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:34.865270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 481/932] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:34 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:35.207622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51058e8f-a556-45f3-9475-106a018ddf1c None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:35.208660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51058e8f-a556-45f3-9475-106a018ddf1c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:35.209166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51058e8f-a556-45f3-9475-106a018ddf1c None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:35.210960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 452/933] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:35.216938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ce8f6f-bcf8-49f6-8319-462fb25734de None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:35.217537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ce8f6f-bcf8-49f6-8319-462fb25734de None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:35.218057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ce8f6f-bcf8-49f6-8319-462fb25734de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:35.242042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-67ce8f6f-bcf8-49f6-8319-462fb25734de None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:35.244611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-67ce8f6f-bcf8-49f6-8319-462fb25734de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:35.275640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:35.322759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51058e8f-a556-45f3-9475-106a018ddf1c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:35.366646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 482/934] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:35.473692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:35.482127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51058e8f-a556-45f3-9475-106a018ddf1c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:35.519619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:35.520320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:35.520655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:35.520924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:35.532942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:35.538048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:35.539152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:35.539975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:35.543671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 453/935] 10.222.0.52 () {56 vars in 1120 bytes} [Sat Aug 27 02:09:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:35.546102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ce8f6f-bcf8-49f6-8319-462fb25734de None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:35.696967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ce8f6f-bcf8-49f6-8319-462fb25734de None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:35.723966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:35.747752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:35.748954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:35.749742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:35.750506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:35.752037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:35.752837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:35.755329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:35.765634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 483/936] 10.222.0.52 () {56 vars in 1115 bytes} [Sat Aug 27 02:09:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:35.778064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:35.820259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:35.889937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6060f3-bd6c-4242-b582-ad0679cdd946 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:35.926600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570a01c8-ee38-41fd-8c2c-a9fed1fcb15b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:35.954933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:35.956879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:35.958101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:35.959073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:35.960610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:35.964841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:35.967095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:35.967880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:35.968250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:35.968612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:35.983960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:35.985222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 27 02:09:35.986219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:35.988716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 454/937] 10.222.0.52 () {56 vars in 1132 bytes} [Sat Aug 27 02:09:35 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:35.998138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:35.998810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:36.006876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:36.025763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:36.034847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:36.036678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:36.038977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:36.038977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:36.038977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:36.042321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 484/938] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:36.053870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:36.178888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eafb16-46cd-4757-ac26-65e863a1ed8e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:36.193552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:36.235022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e28a66-3239-42d3-a164-86fa29fa2dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:36.235705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e28a66-3239-42d3-a164-86fa29fa2dd2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.236072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e28a66-3239-42d3-a164-86fa29fa2dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.236349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e28a66-3239-42d3-a164-86fa29fa2dd2 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.237452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e28a66-3239-42d3-a164-86fa29fa2dd2 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:36.237732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e28a66-3239-42d3-a164-86fa29fa2dd2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:36.239719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e28a66-3239-42d3-a164-86fa29fa2dd2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:36.246554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 455/939] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:35 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:36.246786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8d02c-a098-4da9-8c38-3a5bf5ee8669 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:36.247771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d3dde-876e-4a59-945b-9b0af4437105 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d3dde-876e-4a59-945b-9b0af4437105 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.248316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d3dde-876e-4a59-945b-9b0af4437105 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.249812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 456/940] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:36 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:36.256532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d3dde-876e-4a59-945b-9b0af4437105 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:36.314501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868b1abd-d62c-448f-bf63-d4845323a1d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:36.315542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868b1abd-d62c-448f-bf63-d4845323a1d8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.316172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868b1abd-d62c-448f-bf63-d4845323a1d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.316787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868b1abd-d62c-448f-bf63-d4845323a1d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.317491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868b1abd-d62c-448f-bf63-d4845323a1d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:36.317865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868b1abd-d62c-448f-bf63-d4845323a1d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:36.319290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868b1abd-d62c-448f-bf63-d4845323a1d8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:36.341973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 485/941] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:36.347968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95bf20c-8e63-4d06-a85f-713e715aea50 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.348276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95bf20c-8e63-4d06-a85f-713e715aea50 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.348562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95bf20c-8e63-4d06-a85f-713e715aea50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.360950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:36.369269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d95bf20c-8e63-4d06-a85f-713e715aea50 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:36.370500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d95bf20c-8e63-4d06-a85f-713e715aea50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:36.387478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d3dde-876e-4a59-945b-9b0af4437105 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:36.451793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:36.453073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.453715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.454192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.455712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:36.456293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:36.458942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:36.472696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 457/942] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:36 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:36.472696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:36.506580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 486/943] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:36 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:36.507967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95bf20c-8e63-4d06-a85f-713e715aea50 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:36.585655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:36.620797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df53b8-7af1-441b-9337-9329a4eb2269 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:36.640196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95bf20c-8e63-4d06-a85f-713e715aea50 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:36.675014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8755c565-7812-4291-b230-3253638a77ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:36.676092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755c565-7812-4291-b230-3253638a77ff None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.676741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755c565-7812-4291-b230-3253638a77ff None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.677242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755c565-7812-4291-b230-3253638a77ff None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.686322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8755c565-7812-4291-b230-3253638a77ff None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:36.686680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:36.687641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.688106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.688572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.689027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8755c565-7812-4291-b230-3253638a77ff None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:36.690164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:36.693233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8755c565-7812-4291-b230-3253638a77ff None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:36.702872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:36.703939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:36.704861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:36.713172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 458/944] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:09:36 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:36.716671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8755c565-7812-4291-b230-3253638a77ff None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:36.729143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:36.729557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 487/945] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:36 2022] POST /identity/v3/endpoints => generated 347 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:36.853309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:36.900169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8755c565-7812-4291-b230-3253638a77ff None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:36.963249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:36.964266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:36.964535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:36.964806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:36.965583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:36.965884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:36.967313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:36.990875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 459/946] 10.222.0.52 () {56 vars in 1149 bytes} [Sat Aug 27 02:09:36 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:36.996649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.031485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99ff0d9-43c0-4497-8187-504ba967d410 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:37.057387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:37.061189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:37.089222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7858f54-f965-4fd1-bcde-2ebe27b2fbfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:37.090647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7858f54-f965-4fd1-bcde-2ebe27b2fbfb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.091012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7858f54-f965-4fd1-bcde-2ebe27b2fbfb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.091824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7858f54-f965-4fd1-bcde-2ebe27b2fbfb None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.092986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7858f54-f965-4fd1-bcde-2ebe27b2fbfb None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:37.095323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7858f54-f965-4fd1-bcde-2ebe27b2fbfb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:37.117866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908cce59-41f6-4dd3-8319-330373753afd None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-c7858f54-f965-4fd1-bcde-2ebe27b2fbfb None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:37.118947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:37.120655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:37.122132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 488/947] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:36 2022] GET /identity/v3/services/object-store => generated 93 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:37.123571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff88f92-0ec7-4f8a-ac19-3f953ad82caa None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.123571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff88f92-0ec7-4f8a-ac19-3f953ad82caa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.123806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff88f92-0ec7-4f8a-ac19-3f953ad82caa None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.132461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 489/948] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:37.135317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff88f92-0ec7-4f8a-ac19-3f953ad82caa None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.175522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b55a6c-717b-482f-8391-7f5091f83361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:37.176168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b55a6c-717b-482f-8391-7f5091f83361 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.176531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b55a6c-717b-482f-8391-7f5091f83361 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.176805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b55a6c-717b-482f-8391-7f5091f83361 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.188892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b55a6c-717b-482f-8391-7f5091f83361 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:37.192519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b55a6c-717b-482f-8391-7f5091f83361 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-58b55a6c-717b-482f-8391-7f5091f83361 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:37.212776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:37.213881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:37.214765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:37.214765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:37.214765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:37.221502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 460/949] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:37.221502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096818f3-46ab-4a0b-ae1a-08223ef8f14e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.221502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096818f3-46ab-4a0b-ae1a-08223ef8f14e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.221502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096818f3-46ab-4a0b-ae1a-08223ef8f14e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.238818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-096818f3-46ab-4a0b-ae1a-08223ef8f14e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:37.240049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-096818f3-46ab-4a0b-ae1a-08223ef8f14e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:37.285597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff88f92-0ec7-4f8a-ac19-3f953ad82caa None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:37.329256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:37.330299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.330560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.330832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.337205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 461/950] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:37.337441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:37.337441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:37.337707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096818f3-46ab-4a0b-ae1a-08223ef8f14e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.337847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:37.349170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 490/951] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:37 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:37.351868 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.393916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:37.457568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:37.458764 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096818f3-46ab-4a0b-ae1a-08223ef8f14e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:37.497510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:37.498157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.498435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.498708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.499379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:37.499685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:37.501163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:37.514589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 462/952] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:37.519758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bac87c2-b8f4-4264-bae2-14b48fdb1333 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:37.523955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.569358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:37.570349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.570873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.571392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.572143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:37.572517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:37.574255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:37.585502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 491/953] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:37 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:37.587284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.689777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e635cc1-04f2-440f-b797-52242137de3d None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:37.692056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:37.744635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:37.745656 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.746116 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.746528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.754563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:37.755984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:37.768084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:37.769778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 463/954] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:37.771416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.777954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7584cb4-9058-495b-83cf-9b7ca8c617a6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:37.790065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:37.833043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:37.833903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:37.834344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:37.834794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:37.835770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:37.837501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:37.868099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 492/955] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:37 2022] POST /identity/v3/endpoints => generated 349 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:37.882242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:37.970200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47853f5f-2f62-4e14-8be7-65884a51fd58 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:38.024830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:38.025688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:38.026067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:38.026521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:38.027478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:38.034798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:38.038932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING keystone.server.flask.common [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 27 02:09:38.057244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.sql.core [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 27 02:09:38.057244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 27 02:09:38.057244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [parameters: {'user_id': 'b03e1bcab0464eafb8d04b2c81237d23', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 27 02:09:38.057244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81746) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 27 02:09:38.093933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c1c12e-28f5-4a1b-ab10-d89d0bc3f871 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application result = self._query(query) Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application conn.query(q) Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application result.read() Aug 27 02:09:38.095761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 27 02:09:38.096722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application next(self.gen) Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application yield resource Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application next(self.gen) Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self.session.rollback() Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self.force_reraise() Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise self.value Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application session.commit() Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 27 02:09:38.097792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._prepare_impl() Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self.session.flush() Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._flush(objects) Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application compat.raise_( Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application flush_context.execute() Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rec.execute(self) Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 27 02:09:38.098969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 27 02:09:38.100106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application result = self._query(query) Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application conn.query(q) Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application result.read() Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 27 02:09:38.100968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [parameters: {'user_id': 'b03e1bcab0464eafb8d04b2c81237d23', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:38.101857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 27 02:09:38.102665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 27 02:09:38.103590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 27 02:09:38.103590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 27 02:09:38.103590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 27 02:09:38.103590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 27 02:09:38.103590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 27 02:09:38.103590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:38.103590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 464/956] 10.222.0.52 () {60 vars in 1131 bytes} [Sat Aug 27 02:09:37 2022] POST /identity/v3/users => generated 160 bytes in 328 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:38.107642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:38.153320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:38.159353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:38.160018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:38.160314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:38.160593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:38.161963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:38.162338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:38.164643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:38.185670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 493/957] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:38.196181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:38.253760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:38.317599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f92112-4927-4e56-9d8f-6fe9bfc1ff06 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:38.369366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:38.370108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:38.370438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:38.370764 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:38.382653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:38.384865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:38.388837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106761ed-e629-4613-b665-07a868e526c7 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:38.403061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:38.405151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:38.407632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 465/958] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:09:38 2022] GET /identity/v3/users/cinder => generated 84 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:38.411867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:38.471060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91413bd9-4db1-4ce9-955d-624d0494a27e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:38.472379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91413bd9-4db1-4ce9-955d-624d0494a27e None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:38.472939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91413bd9-4db1-4ce9-955d-624d0494a27e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:38.473549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91413bd9-4db1-4ce9-955d-624d0494a27e None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/1a14c9fa11f94a3ab0c4d32f6f8f0931/roles/58422e7671624a43a2f7a18f07bb7506` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:38.478085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91413bd9-4db1-4ce9-955d-624d0494a27e None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=1a14c9fa11f94a3ab0c4d32f6f8f0931, role_id=58422e7671624a43a2f7a18f07bb7506)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:38.481550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91413bd9-4db1-4ce9-955d-624d0494a27e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:38.514341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 494/959] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:09:38 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/1a14c9fa11f94a3ab0c4d32f6f8f0931/roles/58422e7671624a43a2f7a18f07bb7506 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:09:38.541732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:38.576962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe8437b-040e-4e0e-9355-e26be71964b6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:38.623104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4f44ae-5136-4eed-b9f8-4db8e00f0aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:38.624220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4f44ae-5136-4eed-b9f8-4db8e00f0aa9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:38.624763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4f44ae-5136-4eed-b9f8-4db8e00f0aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:38.625275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4f44ae-5136-4eed-b9f8-4db8e00f0aa9 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:38.626418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4f44ae-5136-4eed-b9f8-4db8e00f0aa9 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:38.627110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4f44ae-5136-4eed-b9f8-4db8e00f0aa9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:38.628853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4f44ae-5136-4eed-b9f8-4db8e00f0aa9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:38.678760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 466/960] 10.222.0.52 () {56 vars in 1101 bytes} [Sat Aug 27 02:09:38 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:39.342541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df54c5-4c69-428a-91bb-d9cb601982aa None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:39.343487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df54c5-4c69-428a-91bb-d9cb601982aa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:39.343883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df54c5-4c69-428a-91bb-d9cb601982aa None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:39.345630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 495/961] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:39.352373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc1802-86d4-479f-945e-372900e6822b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:39.353504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc1802-86d4-479f-945e-372900e6822b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:39.354028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc1802-86d4-479f-945e-372900e6822b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:39.373593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-fdfc1802-86d4-479f-945e-372900e6822b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:39.375214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fdfc1802-86d4-479f-945e-372900e6822b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:39.456869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 467/962] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:39 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:39.848546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df54c5-4c69-428a-91bb-d9cb601982aa None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:39.971309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:39.993679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df54c5-4c69-428a-91bb-d9cb601982aa None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:40.031008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:40.031653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:40.032102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:40.033029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:40.033029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:40.034707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:40.048646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:40.051595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:40.053060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:40.057000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 496/963] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:39 2022] GET /identity/v3/services/compute => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:40.060922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfc1802-86d4-479f-945e-372900e6822b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:40.170465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:40.240486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfc1802-86d4-479f-945e-372900e6822b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:40.297098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a14ae8-8ff2-4b40-bfc3-52432f659ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:40.298163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a14ae8-8ff2-4b40-bfc3-52432f659ca1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:40.298713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a14ae8-8ff2-4b40-bfc3-52432f659ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:40.299283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a14ae8-8ff2-4b40-bfc3-52432f659ca1 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:40.300511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a14ae8-8ff2-4b40-bfc3-52432f659ca1 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:40.301282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a14ae8-8ff2-4b40-bfc3-52432f659ca1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:40.303008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a14ae8-8ff2-4b40-bfc3-52432f659ca1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:40.317491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 468/964] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:40 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:40.321755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:40.424020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a88522-9573-4923-b485-baf320b5fa1e None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:40.424571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a88522-9573-4923-b485-baf320b5fa1e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:40.424829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a88522-9573-4923-b485-baf320b5fa1e None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:40.427114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 469/965] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:40.435234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901c298b-7059-429a-89f6-b240ddc842af None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:40.435643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901c298b-7059-429a-89f6-b240ddc842af None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:40.436049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901c298b-7059-429a-89f6-b240ddc842af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:40.461430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-901c298b-7059-429a-89f6-b240ddc842af None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:40.463029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-901c298b-7059-429a-89f6-b240ddc842af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:40.496516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b077b997-e89a-451c-81c3-beb6f0dee887 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:40.555403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:40.556385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:40.556863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:40.557246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:40.558087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:40.558539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:40.560752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:40.569584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 497/966] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:40 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:40.582468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 470/967] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:40 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:40.890393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:40.911876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:40.954671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc81423-2fa8-463c-bf93-9a8783612ad7 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:40.955108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc81423-2fa8-463c-bf93-9a8783612ad7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:40.955541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc81423-2fa8-463c-bf93-9a8783612ad7 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:40.957792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 471/968] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:40.967004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2b2c1b-ac00-4ec6-980b-a3fc4f1d0a1a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:40.967449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2b2c1b-ac00-4ec6-980b-a3fc4f1d0a1a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:40.967880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2b2c1b-ac00-4ec6-980b-a3fc4f1d0a1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:40.996926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-cb2b2c1b-ac00-4ec6-980b-a3fc4f1d0a1a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:40.999135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-cb2b2c1b-ac00-4ec6-980b-a3fc4f1d0a1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:41.065025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a521d31-e283-4e53-b7c5-00c4cd3c6872 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:41.087364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:41.117655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 472/969] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:40 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:41.145613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:41.147600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:41.147997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:41.148466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:41.149191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:41.151337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:41.167299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:41.168654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:41.169934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:41.174188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 498/970] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:40 2022] GET /identity/v3/services/object-store => generated 93 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:41.184321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2b2c1b-ac00-4ec6-980b-a3fc4f1d0a1a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:41.355917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2b2c1b-ac00-4ec6-980b-a3fc4f1d0a1a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:41.408406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:41.409279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:41.409616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:41.409892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:41.410620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:41.410952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:41.412702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:41.419564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 473/971] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:41.427997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:41.526962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:41.564201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:41.611789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609d6f2a-7acb-4392-ba4d-9e457c5ae1f3 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:41.617732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:41.679420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:41.680364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:41.680659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:41.681063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:41.682090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:41.682705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:41.684999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:41.692833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 499/972] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:41.693108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853db89b-a96e-4568-af51-6aa5d06682b3 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:41.701580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:41.745721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:41.747730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:41.748228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:41.748711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:41.767759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:41.780174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:41.783130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:41.784255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 474/973] 10.222.0.52 () {56 vars in 1120 bytes} [Sat Aug 27 02:09:41 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:41.790998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:41.873603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5be793c-1e03-4ac1-bce6-dc4cfb0b149e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:41.924447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:41.925409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:41.925877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:41.926340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:41.927175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] RBAC: Adding query filter params (interface=admin, service_id=124ee2ed9e84424b8220a84c5ab1aba6, region_id=RegionOne) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:41.927485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:41.928876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:41.948046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 500/974] 10.222.0.52 () {56 vars in 1249 bytes} [Sat Aug 27 02:09:41 2022] GET /identity/v3/endpoints?service_id=124ee2ed9e84424b8220a84c5ab1aba6&interface=admin®ion_id=RegionOne => generated 193 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:41.949902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:41.957180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:41.988408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a469485-fe5a-46af-a6f5-e077dfe7f8ea None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:42.015120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:42.049088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:42.050252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:42.050699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:42.051175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:42.052439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:42.052923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:42.055540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:42.070115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 475/975] 10.222.0.52 () {56 vars in 1115 bytes} [Sat Aug 27 02:09:41 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:42.076605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:42.118660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98dc8d1-b5f7-48d8-b904-a956290ecaa5 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:42.176981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:42.177932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:42.178392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:42.178669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:42.179293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:42.180880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:42.201275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 501/976] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:41 2022] GET /identity/v3/services => generated 1497 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:09:42.252505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50b090-4fd4-4e4a-bacb-460bc0277536 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:42.301508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:42.302176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:42.302456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:42.302779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:42.317467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:42.319584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:42.343712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:42.346199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:42.348047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:42.348047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:42.348047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:42.348047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 476/977] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:42.355259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:42.449705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:42.512900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3aa211b-a442-42f4-8127-855e242a5d0c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:42.577849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93e277d-68cd-44ff-9049-ac7bfd443e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:42.578862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e277d-68cd-44ff-9049-ac7bfd443e3e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:42.579337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e277d-68cd-44ff-9049-ac7bfd443e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:42.580114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e277d-68cd-44ff-9049-ac7bfd443e3e None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:42.580897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93e277d-68cd-44ff-9049-ac7bfd443e3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:42.581475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93e277d-68cd-44ff-9049-ac7bfd443e3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:42.583908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93e277d-68cd-44ff-9049-ac7bfd443e3e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:42.611178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 502/978] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:42.612593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:42.653963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:42.753395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a0edd-e1cd-47bf-b65a-d8e1c0350ce1 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:42.753739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a0edd-e1cd-47bf-b65a-d8e1c0350ce1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:42.753942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a0edd-e1cd-47bf-b65a-d8e1c0350ce1 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:42.755445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 503/979] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:42.760852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8efdd4-47cc-4d4c-b42b-bd198a8bec62 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:42.761169 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8efdd4-47cc-4d4c-b42b-bd198a8bec62 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:42.761422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8efdd4-47cc-4d4c-b42b-bd198a8bec62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:42.781844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5d8efdd4-47cc-4d4c-b42b-bd198a8bec62 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:42.783129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5d8efdd4-47cc-4d4c-b42b-bd198a8bec62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:42.788151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40270931-be9a-4da5-877a-d1e1d26d5295 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:42.841561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:42.842523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:42.842977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:42.843444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:42.852971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:42.854723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:42.868348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:42.869606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:42.870603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 477/980] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:09:42 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:42.875599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:42.877033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 504/981] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:43.022107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:43.040112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4900766-a2eb-4a61-b18d-b192a9fd0c9e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:43.089589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:43.090200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:43.090471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:43.090739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:43.091490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:43.091796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:43.093244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:43.119655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 478/982] 10.222.0.52 () {56 vars in 1149 bytes} [Sat Aug 27 02:09:42 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:43.128887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8efdd4-47cc-4d4c-b42b-bd198a8bec62 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:43.142633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:43.221691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:43.284622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8efdd4-47cc-4d4c-b42b-bd198a8bec62 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:43.301356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1f840-eaae-4edc-bce8-8ea89a6c71b8 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:43.341180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:43.342145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:43.342626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:43.343090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:43.344064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:43.344903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:43.345698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:43.345698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:43.346091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:43.347459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:43.369170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 479/983] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:43 2022] POST /identity/v3/services => generated 242 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:09:43.443874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:43.445391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:43.538521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:43.600098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 505/984] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:43 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 472 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:09:43.606751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:43.627891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:43.753924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37c4ea-d8db-4ca3-90cf-549e37bd15ec None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:43.793732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:43.794469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:43.794741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:43.795004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:43.807376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:43.809312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:43.826894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:43.828842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:43.830407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:43.830407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:43.830407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:43.830407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 480/985] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:43.837626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:43.973535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6502f16-57a9-4b23-a911-165307810b93 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:44.010774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:44.011414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:44.011687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:44.011918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:44.012678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:44.012955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:44.014353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:44.037177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 506/986] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:44.038974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:44.121569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:44.177075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc4c95-991c-4641-9955-e785bd2cba87 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:44.226173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf63ef9c-bb14-47c3-958c-a68ffac3acbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:44.226913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63ef9c-bb14-47c3-958c-a68ffac3acbf None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:44.227310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63ef9c-bb14-47c3-958c-a68ffac3acbf None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:44.227698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63ef9c-bb14-47c3-958c-a68ffac3acbf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:44.236409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf63ef9c-bb14-47c3-958c-a68ffac3acbf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:44.238240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf63ef9c-bb14-47c3-958c-a68ffac3acbf None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-bf63ef9c-bb14-47c3-958c-a68ffac3acbf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:44.262399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:44.264178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:44.265935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:44.269381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 481/987] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:44.274827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:44.311029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:44.464285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe465737-a6b2-4fab-ad96-87b97914b0e1 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:44.496305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89057384-08a8-4d24-bd10-a6f52d360bb3 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:44.496699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89057384-08a8-4d24-bd10-a6f52d360bb3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:44.497103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89057384-08a8-4d24-bd10-a6f52d360bb3 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:44.502618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 482/988] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:44 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:44.509162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb5f10-988c-400a-865c-09c801c48594 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:44.509454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb5f10-988c-400a-865c-09c801c48594 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:44.509771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb5f10-988c-400a-865c-09c801c48594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:44.524555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:44.525382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:44.526418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:44.526846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:44.528038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:44.528541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:44.530186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:44.532570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c3fb5f10-988c-400a-865c-09c801c48594 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:44.533614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c3fb5f10-988c-400a-865c-09c801c48594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:44.546134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 507/989] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:44.554667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:44.633356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 483/990] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:44 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:44.683780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:44.727856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525040ca-c227-4ddb-b34c-17c4341e5302 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:44.782499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:44.784152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:44.784585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:44.785037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:44.786500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] RBAC: Adding query filter params (role.id=58422e7671624a43a2f7a18f07bb7506, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:44.787010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:44.789569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:44.799682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 508/991] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:44 2022] GET /identity/v3/role_assignments?user.id=1a14c9fa11f94a3ab0c4d32f6f8f0931&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=58422e7671624a43a2f7a18f07bb7506 => generated 608 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:44.906576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb5f10-988c-400a-865c-09c801c48594 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:44.959108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:45.019687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb5f10-988c-400a-865c-09c801c48594 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:45.060443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:45.060997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:45.061311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:45.061586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:45.062208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:45.063710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:45.076515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:45.078577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:45.081196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 484/992] 10.222.0.52 () {56 vars in 1126 bytes} [Sat Aug 27 02:09:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:45.084214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:45.211504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f14ee2-79ab-4d2f-b385-7aff0576b02b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:45.251993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:45.252613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:45.252895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:45.253232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:45.253879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:45.254181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:45.256031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:45.262893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 509/993] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:45 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:45.269704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:45.416424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8896c1fb-29bb-47ca-9400-8ed3d093ef97 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:45.432857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:45.480196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ce9ec8-b53a-4d38-9b57-7a067ef0deb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:45.480808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce9ec8-b53a-4d38-9b57-7a067ef0deb6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:45.481084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce9ec8-b53a-4d38-9b57-7a067ef0deb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:45.481384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce9ec8-b53a-4d38-9b57-7a067ef0deb6 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:45.483819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ce9ec8-b53a-4d38-9b57-7a067ef0deb6 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:45.483819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ce9ec8-b53a-4d38-9b57-7a067ef0deb6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:45.483819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ce9ec8-b53a-4d38-9b57-7a067ef0deb6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:45.493959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 485/994] 10.222.0.52 () {56 vars in 1122 bytes} [Sat Aug 27 02:09:45 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:45.500607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:45.513873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:45.613420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8515e498-279b-4faa-a2c4-47948f421e96 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:45.613676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8515e498-279b-4faa-a2c4-47948f421e96 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:45.614030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8515e498-279b-4faa-a2c4-47948f421e96 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:45.615586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 486/995] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:45.620857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e729477-da8f-4b98-b630-e43fc2e02971 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:45.621215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e729477-da8f-4b98-b630-e43fc2e02971 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:45.621564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e729477-da8f-4b98-b630-e43fc2e02971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:45.640732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7e729477-da8f-4b98-b630-e43fc2e02971 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:45.642093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7e729477-da8f-4b98-b630-e43fc2e02971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:45.681342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d920cd5-2263-408b-b3ef-0784ab43ce4a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:45.741630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 487/996] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:45 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:45.744454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:45.745488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:45.745933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:45.746343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:45.747473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:45.750079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:45.789158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 510/997] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:45 2022] POST /identity/v3/endpoints => generated 325 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:09:46.014495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e729477-da8f-4b98-b630-e43fc2e02971 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:46.179471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e729477-da8f-4b98-b630-e43fc2e02971 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:46.183598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:46.233205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:46.233851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:46.234152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:46.234419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:46.235012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:46.237076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:46.246923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:46.248048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:46.249290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 488/998] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:46 2022] GET /identity/v3/services/compute => generated 88 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:46.260624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:46.368711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:46.420885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b42dca8-050c-4dd6-940b-ec7e7fab6395 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:46.473631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:46.474487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:46.474763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:46.475034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:46.475755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] RBAC: Adding query filter params (name=compute) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:46.476055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:46.477605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:46.490077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 511/999] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:46 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:46.496180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:46.628666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286cf47-7f9c-4116-86ae-544dc0eca394 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:46.691398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:46.692777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:46.693310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:46.693789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:46.694917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:46.695432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:46.697870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:46.707873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 489/1000] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:46 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:46.717183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:46.895058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271a89ac-ab1f-4364-8b52-3b946a4b64fb None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:46.935670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:46.948611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16968d44-63c7-4cad-8164-9caf7048074e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:46.949689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16968d44-63c7-4cad-8164-9caf7048074e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:46.950175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16968d44-63c7-4cad-8164-9caf7048074e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:46.950670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16968d44-63c7-4cad-8164-9caf7048074e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:46.951876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16968d44-63c7-4cad-8164-9caf7048074e None admin] RBAC: Adding query filter params (interface=public, service_id=19cc6d458542401294b4d6f43c703de2, region_id=RegionOne) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:46.952290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16968d44-63c7-4cad-8164-9caf7048074e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:46.954142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16968d44-63c7-4cad-8164-9caf7048074e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:46.967310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 512/1001] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:09:46 2022] GET /identity/v3/endpoints?service_id=19cc6d458542401294b4d6f43c703de2&interface=public®ion_id=RegionOne => generated 194 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:46.969623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:47.010268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:47.046952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1fc72-fbb3-48fb-92a2-28bff75781ad None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:47.047369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1fc72-fbb3-48fb-92a2-28bff75781ad None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:47.047648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1fc72-fbb3-48fb-92a2-28bff75781ad None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:47.049461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 513/1002] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:09:47.055092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda38d0-b12f-403b-8ccf-d76a4202428c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:47.055376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda38d0-b12f-403b-8ccf-d76a4202428c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:47.055627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda38d0-b12f-403b-8ccf-d76a4202428c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:47.076841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3fda38d0-b12f-403b-8ccf-d76a4202428c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:47.078709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3fda38d0-b12f-403b-8ccf-d76a4202428c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:47.137220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744d658b-5a26-42b6-8b2d-ce5e723f8623 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:47.161365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 514/1003] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:47 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:47.199386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:47.200465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:47.200943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:47.201473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:47.202644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:47.205373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:47.225152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 490/1004] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:46 2022] GET /identity/v3/services => generated 1727 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:09:47.457746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda38d0-b12f-403b-8ccf-d76a4202428c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:47.615381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda38d0-b12f-403b-8ccf-d76a4202428c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:47.651095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:47.669823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:47.670478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:47.670811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:47.671081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] PATH_INFO: `/v3/services/image` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:47.671703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:47.673722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:47.684660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:47.686765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:47.688306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 515/1005] 10.222.0.52 () {56 vars in 1105 bytes} [Sat Aug 27 02:09:47 2022] GET /identity/v3/services/image => generated 86 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:47.699482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:47.815218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:47.827797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a553ea5-027c-4072-8518-a6900d8e002e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:47.867934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5b0816-d1ad-4640-99a5-6ffc93481a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:47.868706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5b0816-d1ad-4640-99a5-6ffc93481a79 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:47.869002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5b0816-d1ad-4640-99a5-6ffc93481a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:47.869289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5b0816-d1ad-4640-99a5-6ffc93481a79 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:47.870175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5b0816-d1ad-4640-99a5-6ffc93481a79 None admin] RBAC: Adding query filter params (name=image) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:47.870531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5b0816-d1ad-4640-99a5-6ffc93481a79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:47.872293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5b0816-d1ad-4640-99a5-6ffc93481a79 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:47.883420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 491/1006] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:47 2022] GET /identity/v3/services?name=image => generated 122 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:47.888201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:47.988423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa26a3e9-afc4-46e4-a4f0-b3c8124864d5 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:47.988628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa26a3e9-afc4-46e4-a4f0-b3c8124864d5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:47.988999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa26a3e9-afc4-46e4-a4f0-b3c8124864d5 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:47.990634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 492/1007] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:47.996353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18884906-1f35-4264-80d1-c80b5f00eb6a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:47.996555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18884906-1f35-4264-80d1-c80b5f00eb6a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:47.996878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18884906-1f35-4264-80d1-c80b5f00eb6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:48.017734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-18884906-1f35-4264-80d1-c80b5f00eb6a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:48.019056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-18884906-1f35-4264-80d1-c80b5f00eb6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:48.049773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c37fecc-a1b2-4f4c-ab3c-5d0ca08d266c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:48.094369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:48.095370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:48.095847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:48.096352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:48.097348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] RBAC: Adding query filter params (type=image) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:48.097833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:48.099581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:48.113167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 516/1008] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:47 2022] GET /identity/v3/services?type=image => generated 122 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:48.116196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 493/1009] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:47 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:48.476311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:48.570906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:48.666468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640b7782-6885-40d8-9987-344648dc6af8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:48.714350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01fc6eb-0ebf-45de-8ed6-d6bf7c84e2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:48.716205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01fc6eb-0ebf-45de-8ed6-d6bf7c84e2fa None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:48.716462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01fc6eb-0ebf-45de-8ed6-d6bf7c84e2fa None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:48.716839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01fc6eb-0ebf-45de-8ed6-d6bf7c84e2fa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:48.718750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01fc6eb-0ebf-45de-8ed6-d6bf7c84e2fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:48.721688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01fc6eb-0ebf-45de-8ed6-d6bf7c84e2fa None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:48.725209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 517/1010] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:48.736438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18884906-1f35-4264-80d1-c80b5f00eb6a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:48.883387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:48.911103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18884906-1f35-4264-80d1-c80b5f00eb6a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:48.977418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35502710-b3c6-4670-83fe-7c2f30aa4b83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:48.978436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35502710-b3c6-4670-83fe-7c2f30aa4b83 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:48.978918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35502710-b3c6-4670-83fe-7c2f30aa4b83 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:48.979393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35502710-b3c6-4670-83fe-7c2f30aa4b83 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:48.980678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35502710-b3c6-4670-83fe-7c2f30aa4b83 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:48.983372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35502710-b3c6-4670-83fe-7c2f30aa4b83 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:49.037266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 494/1011] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:48 2022] POST /identity/v3/projects => generated 311 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:09:49.388926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed12aa7-50ba-4258-ba8b-e4df05be7670 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:49.389808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed12aa7-50ba-4258-ba8b-e4df05be7670 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:49.389808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed12aa7-50ba-4258-ba8b-e4df05be7670 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:49.391217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 518/1012] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:49.399870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6d5db-f5c1-4a5d-8626-bf2fc6a9de3f None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:49.400190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6d5db-f5c1-4a5d-8626-bf2fc6a9de3f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:49.400460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6d5db-f5c1-4a5d-8626-bf2fc6a9de3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:49.427323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5eb6d5db-f5c1-4a5d-8626-bf2fc6a9de3f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:49.429295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5eb6d5db-f5c1-4a5d-8626-bf2fc6a9de3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:49.535477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 495/1013] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:49 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:49.848842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed12aa7-50ba-4258-ba8b-e4df05be7670 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:50.008018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed12aa7-50ba-4258-ba8b-e4df05be7670 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:50.031072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:50.052866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:50.053914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:50.054368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:50.054804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:50.055777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:50.058196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:50.071433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:50.073746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:50.074611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 519/1014] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:49 2022] GET /identity/v3/services/compute => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:50.082002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb6d5db-f5c1-4a5d-8626-bf2fc6a9de3f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:50.181006 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:50.236825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb6d5db-f5c1-4a5d-8626-bf2fc6a9de3f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:50.280489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0233e2e-e606-41ca-b16b-42e7be2f6779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:50.281386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0233e2e-e606-41ca-b16b-42e7be2f6779 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:50.281928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0233e2e-e606-41ca-b16b-42e7be2f6779 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:50.282517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0233e2e-e606-41ca-b16b-42e7be2f6779 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:50.283472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0233e2e-e606-41ca-b16b-42e7be2f6779 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:50.283917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0233e2e-e606-41ca-b16b-42e7be2f6779 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:50.285906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0233e2e-e606-41ca-b16b-42e7be2f6779 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:50.293978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 496/1015] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:50 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:50.303595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:50.332988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4f006-208e-4c79-9a44-cfc0f4d94859 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:50.333528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4f006-208e-4c79-9a44-cfc0f4d94859 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:50.334021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4f006-208e-4c79-9a44-cfc0f4d94859 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:50.336655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 497/1016] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:50.342673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce79085b-edc4-4974-9526-63600d3a4545 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:50.342959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce79085b-edc4-4974-9526-63600d3a4545 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:50.343213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce79085b-edc4-4974-9526-63600d3a4545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:50.363309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ce79085b-edc4-4974-9526-63600d3a4545 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:50.365222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ce79085b-edc4-4974-9526-63600d3a4545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:50.454961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db086a89-54ab-4f25-886b-7080b3ac85f4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:50.460931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 498/1017] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:50 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:50.507565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:50.508199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:50.508475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:50.508744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:50.509542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:50.509871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:50.511353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:50.521484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 520/1018] 10.222.0.52 () {56 vars in 1112 bytes} [Sat Aug 27 02:09:50 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:50.524192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce79085b-edc4-4974-9526-63600d3a4545 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:50.624301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:50.684928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce79085b-edc4-4974-9526-63600d3a4545 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:50.740075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca14054-0e58-45bd-8b8e-3167df78868c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:50.740690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca14054-0e58-45bd-8b8e-3167df78868c None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:50.740968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca14054-0e58-45bd-8b8e-3167df78868c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:50.741247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca14054-0e58-45bd-8b8e-3167df78868c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:50.741942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca14054-0e58-45bd-8b8e-3167df78868c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:50.743309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca14054-0e58-45bd-8b8e-3167df78868c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:50.759531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:50.771059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 499/1019] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:50 2022] POST /identity/v3/endpoints => generated 334 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:50.830114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:50.904542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084cd8ca-2594-44b9-8f30-6a21fd48a083 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:50.963070 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd830b4-0408-4dbd-9677-f8606f50cb96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:50.963687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd830b4-0408-4dbd-9677-f8606f50cb96 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:50.963959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd830b4-0408-4dbd-9677-f8606f50cb96 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:50.964221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd830b4-0408-4dbd-9677-f8606f50cb96 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:50.964877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd830b4-0408-4dbd-9677-f8606f50cb96 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:50.966327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd830b4-0408-4dbd-9677-f8606f50cb96 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:50.984723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 521/1020] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:09:50 2022] POST /identity/v3/services => generated 242 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:09:51.197568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0073f32-f397-47ec-ab66-a3115ae0b312 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:51.198092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0073f32-f397-47ec-ab66-a3115ae0b312 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:51.198318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0073f32-f397-47ec-ab66-a3115ae0b312 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:51.200384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 500/1021] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:51.208477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3c3a4f-f9d9-45e3-a36a-125ff07507f2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:51.208820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3c3a4f-f9d9-45e3-a36a-125ff07507f2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:51.209049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3c3a4f-f9d9-45e3-a36a-125ff07507f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:51.230645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ef3c3a4f-f9d9-45e3-a36a-125ff07507f2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:51.231875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ef3c3a4f-f9d9-45e3-a36a-125ff07507f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:51.335085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 522/1022] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:51 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:51.651183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0073f32-f397-47ec-ab66-a3115ae0b312 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:51.804210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0073f32-f397-47ec-ab66-a3115ae0b312 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:51.827804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:51.851895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df56d06-b23a-4e41-9f0b-0bc3d24edc31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:51.852839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df56d06-b23a-4e41-9f0b-0bc3d24edc31 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:51.853321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df56d06-b23a-4e41-9f0b-0bc3d24edc31 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:51.853828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df56d06-b23a-4e41-9f0b-0bc3d24edc31 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:51.855239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df56d06-b23a-4e41-9f0b-0bc3d24edc31 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:51.857737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df56d06-b23a-4e41-9f0b-0bc3d24edc31 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:51.860891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 501/1023] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:51.869028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3c3a4f-f9d9-45e3-a36a-125ff07507f2 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:51.903870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:52.005738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3c3a4f-f9d9-45e3-a36a-125ff07507f2 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:52.057746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a28551a-94bc-4ce4-9a8f-06a4399fb238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:52.058402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a28551a-94bc-4ce4-9a8f-06a4399fb238 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:52.058672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a28551a-94bc-4ce4-9a8f-06a4399fb238 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:52.058979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a28551a-94bc-4ce4-9a8f-06a4399fb238 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:52.059733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a28551a-94bc-4ce4-9a8f-06a4399fb238 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:52.061216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a28551a-94bc-4ce4-9a8f-06a4399fb238 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:52.090726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 523/1024] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:09:51 2022] POST /identity/v3/users => generated 285 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:53.019131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fdbdb5-c05b-41b4-bef4-aca005a714c5 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:53.019891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fdbdb5-c05b-41b4-bef4-aca005a714c5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:53.019891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fdbdb5-c05b-41b4-bef4-aca005a714c5 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:53.020934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 502/1025] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:53.027573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd38755-dbca-435a-b9e9-eeb801e91928 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:53.028477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd38755-dbca-435a-b9e9-eeb801e91928 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:53.028477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd38755-dbca-435a-b9e9-eeb801e91928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:53.051579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1bd38755-dbca-435a-b9e9-eeb801e91928 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:53.053021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1bd38755-dbca-435a-b9e9-eeb801e91928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:53.155967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 524/1026] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:53 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:09:53.320058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3efbe6f-e826-4505-a29f-d9c52887d529 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:53.320493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3efbe6f-e826-4505-a29f-d9c52887d529 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:53.320853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3efbe6f-e826-4505-a29f-d9c52887d529 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:53.323109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 503/1027] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:53.330529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5270bb90-e49a-459c-b10b-69f685f527d3 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:53.330978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5270bb90-e49a-459c-b10b-69f685f527d3 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:53.331394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5270bb90-e49a-459c-b10b-69f685f527d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:53.354870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5270bb90-e49a-459c-b10b-69f685f527d3 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:53.356869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5270bb90-e49a-459c-b10b-69f685f527d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:53.378948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:53.447421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3efbe6f-e826-4505-a29f-d9c52887d529 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:53.490181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 525/1028] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:53 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:53.635496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3efbe6f-e826-4505-a29f-d9c52887d529 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:53.717103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:53.718154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:53.718633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:53.719121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] PATH_INFO: `/v3/services/image` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:53.720256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:53.722808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:53.740538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:53.741885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:53.746305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 504/1029] 10.222.0.52 () {56 vars in 1105 bytes} [Sat Aug 27 02:09:53 2022] GET /identity/v3/services/image => generated 86 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:53.751122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5270bb90-e49a-459c-b10b-69f685f527d3 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:53.824521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:53.914621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:53.939910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5270bb90-e49a-459c-b10b-69f685f527d3 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:53.991442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:53.992353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:53.992772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:53.993367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:53.994611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] RBAC: Adding query filter params (name=image) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:53.995116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:53.997465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:54.003826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7947d4db-2589-4ef1-b7c8-a95a8cbb1bed None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:54.009530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 526/1030] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:53 2022] GET /identity/v3/services?name=image => generated 122 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:54.011073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:54.030926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:54.052021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:54.053034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.053494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.053938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.071699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:54.079866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:54.081871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:09:54.083958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:54.088672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 505/1031] 10.222.0.52 () {56 vars in 1120 bytes} [Sat Aug 27 02:09:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:54.097333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:54.208314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbed4d-e26c-457f-ae77-54bcf39ef6fc None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:54.272331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:54.273680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.274344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.274977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.275846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] RBAC: Adding query filter params (type=image) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:54.276510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:54.278647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:54.286166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e36f0a-7fd1-401d-b9bc-440b33d0a649 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:54.289857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 527/1032] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:54 2022] GET /identity/v3/services?type=image => generated 350 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:54.296163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:54.305354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:54.346169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:54.347076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.347352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.347599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.348287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:54.348588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:54.349997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:54.364597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 506/1033] 10.222.0.52 () {56 vars in 1115 bytes} [Sat Aug 27 02:09:54 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:54.373287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:54.415563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:54.452759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14743b35-aa36-461e-ab08-c9060c2ade2c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:54.494424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7087e4c-0f7b-4760-a62f-3c91d253c4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:54.494945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7087e4c-0f7b-4760-a62f-3c91d253c4cb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.495217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7087e4c-0f7b-4760-a62f-3c91d253c4cb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.495612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7087e4c-0f7b-4760-a62f-3c91d253c4cb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.496490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7087e4c-0f7b-4760-a62f-3c91d253c4cb None admin] RBAC: Adding query filter params (interface=public, service_id=a566ad6034cb4ad0a1309dc670882ddc, region_id=RegionOne) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:54.496847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7087e4c-0f7b-4760-a62f-3c91d253c4cb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:54.498650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7087e4c-0f7b-4760-a62f-3c91d253c4cb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:54.506731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 528/1034] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:09:54 2022] GET /identity/v3/endpoints?service_id=a566ad6034cb4ad0a1309dc670882ddc&interface=public®ion_id=RegionOne => generated 194 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:54.506935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ab4388-7ada-489e-9c06-644dc6ca7565 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:54.508666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5c4e4-86f0-4c57-ac0c-18c1c998f6ce None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.509105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5c4e4-86f0-4c57-ac0c-18c1c998f6ce None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.509482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5c4e4-86f0-4c57-ac0c-18c1c998f6ce None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.510929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 529/1035] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:54.514335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5c4e4-86f0-4c57-ac0c-18c1c998f6ce None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:54.550758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a489493-0efd-4491-95f0-a6fb59ad0bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:54.551477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a489493-0efd-4491-95f0-a6fb59ad0bea None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.551778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a489493-0efd-4491-95f0-a6fb59ad0bea None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.552100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a489493-0efd-4491-95f0-a6fb59ad0bea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.566007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a489493-0efd-4491-95f0-a6fb59ad0bea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:54.568183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a489493-0efd-4491-95f0-a6fb59ad0bea None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-9a489493-0efd-4491-95f0-a6fb59ad0bea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:54.596547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:54.598570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:54.600452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:54.600452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:54.600452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:54.600452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 507/1036] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:54.603374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f6b0a7-3d49-4ffc-b37d-29562a5607fd None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.603657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f6b0a7-3d49-4ffc-b37d-29562a5607fd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.603998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f6b0a7-3d49-4ffc-b37d-29562a5607fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.626898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-08f6b0a7-3d49-4ffc-b37d-29562a5607fd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:54.628177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-08f6b0a7-3d49-4ffc-b37d-29562a5607fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:54.651655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:54.676235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5c4e4-86f0-4c57-ac0c-18c1c998f6ce None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:54.732236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 508/1037] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:54 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:54.733284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f6b0a7-3d49-4ffc-b37d-29562a5607fd None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:54.748822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:54.750115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.750573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.751223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.752863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:54.755315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:54.768401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 530/1038] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:09:54 2022] GET /identity/v3/services => generated 1957 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:09:54.886708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f6b0a7-3d49-4ffc-b37d-29562a5607fd None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:54.937485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:54.938501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:54.938775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:54.939176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:54.940186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:54.940594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:54.942462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:54.965399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 509/1039] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:54.973023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:54.977648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:55.030972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:55.061451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:55.166194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbfb432-3485-40f7-9d54-25a6d91e80f9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:55.215048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1de430-f60c-4122-a443-07a6681b97eb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:55.225800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:55.226707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:55.227111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:55.228192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:55.240733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:55.242703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:09:55.264636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:55.275369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 531/1040] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:09:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:55.277770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:55.279502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:55.279719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:55.279957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:55.280348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:55.311203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:09:55.319181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:55.325529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:55.327079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 510/1041] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:55.328865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:55.388166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:55.454075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a1f93-c506-479c-9fd0-f2c833cc1fe8 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:55.455570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:55.491602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6077aaed-5772-48e8-b6a6-96b512c12f74 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:55.502538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:55.503230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:55.503570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:55.504040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:55.504926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:55.505267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:55.506721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:55.539847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:55.540454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:55.540725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:55.540992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:55.542094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:55.542094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:55.543439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:55.545332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 532/1042] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:09:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:55.547718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:55.557280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 511/1043] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:09:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:55.565196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:55.708818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d8ed6d-9d26-4099-9f68-a17f935f4cb4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:55.757911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:55.768777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0967c481-67bc-468a-8102-262f55a2ce36 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:55.780009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:55.780803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:55.781261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:55.781650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:55.828634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:55.829658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:55.830102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:55.830541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] PATH_INFO: `/v3/users/177c0f0b47ef45789700def6963b4b1d` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:55.838195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:55.858782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] RBAC: Authorizing `identity:get_user(user_id=177c0f0b47ef45789700def6963b4b1d)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:55.861349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:55.865594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 512/1044] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:09:55 2022] GET /identity/v3/users/177c0f0b47ef45789700def6963b4b1d => generated 285 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:55.872826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:55.910771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:55.913578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:56.052842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efda99bf-cedd-4158-b52b-8ace4aff8f2b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:56.081749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 533/1045] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:55 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:09:56.086755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:56.126265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:56.129334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:56.130619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:56.131238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:56.131741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:56.246666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc067e5d-fad9-4254-bff8-34bd2f949313 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:56.257823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:56.272787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:56.276316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:56.300222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:56.301026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:56.301500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:56.301947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:56.327196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:56.330606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:09:56.354473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:56.356920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:09:56.358534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:09:56.358534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:09:56.358534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:56.358534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 534/1046] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:09:56.362439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:56.435771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 513/1047] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:09:55 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 563 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 27 02:09:56.446157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:56.491372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:56.556029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9ac56-3b16-4902-b3bb-a6e2b8f29014 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:56.597340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:56.612698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d187b9-c316-46cb-b7f3-4eece63be941 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:56.616356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:56.617392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:56.617873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:56.618278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:56.619319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:56.619732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:56.621822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:56.644166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 535/1048] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:09:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:56.653335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:56.682545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:56.683287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:56.683726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:56.684105 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:56.686128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] RBAC: Authorizing `identity:get_project(project_id=2e4b71fc6712404188dff9909f62cbef)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:56.688238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:56.691015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 514/1049] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:09:56 2022] GET /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef => generated 311 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:56.699562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:56.838629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5524285-1b18-4dba-9464-665a8ceb450c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:56.853717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb1b17d-d1c4-4d4d-bde4-c68b50c03c64 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:56.905277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:56.915426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:56.916411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:56.917430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:56.917774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:56.919404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=2e4b71fc6712404188dff9909f62cbef, user.id=177c0f0b47ef45789700def6963b4b1d) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:56.919952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:56.921822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:56.926474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:56.927593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:56.928087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:56.928586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:56.942376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:56.953218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 515/1050] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:56 2022] GET /identity/v3/role_assignments?user.id=177c0f0b47ef45789700def6963b4b1d&scope.project.id=2e4b71fc6712404188dff9909f62cbef&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:56.953493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:09:56.975409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:56.977386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:09:56.979992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:09:56.985401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 536/1051] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:09:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:56.986884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:57.032223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:57.169502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8d4914-3d9f-4cd0-a11c-93bf69d2edf2 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:57.223391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe04c0-6a6c-45e6-aeba-dff2e83360cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:57.224066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe04c0-6a6c-45e6-aeba-dff2e83360cc None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:57.224345 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe04c0-6a6c-45e6-aeba-dff2e83360cc None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:57.224701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe04c0-6a6c-45e6-aeba-dff2e83360cc None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:57.225600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe04c0-6a6c-45e6-aeba-dff2e83360cc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:57.226020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe04c0-6a6c-45e6-aeba-dff2e83360cc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:57.227786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe04c0-6a6c-45e6-aeba-dff2e83360cc None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:57.248558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 516/1052] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:09:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:09:57.250401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:57.336297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37817b2-c448-42fb-adf5-07fe81493702 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:57.336800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37817b2-c448-42fb-adf5-07fe81493702 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:57.337391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37817b2-c448-42fb-adf5-07fe81493702 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:57.339921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 517/1053] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:57.347631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba629eb-7c48-4c0d-af10-6ff69d10019e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:57.348067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba629eb-7c48-4c0d-af10-6ff69d10019e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:57.348516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba629eb-7c48-4c0d-af10-6ff69d10019e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:57.379329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-aba629eb-7c48-4c0d-af10-6ff69d10019e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:57.380933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-aba629eb-7c48-4c0d-af10-6ff69d10019e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:57.452284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbddbaf-230e-4318-ba94-49996d75caf4 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:57.510512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 518/1054] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:57 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:57.515670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:57.516366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:57.516366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:57.516614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:57.517580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] RBAC: Adding query filter params (role.id=58422e7671624a43a2f7a18f07bb7506, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=c83f0d0e2e8047a98fbe114ceb15c6e0) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:57.517735 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:57.519213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:57.529171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 537/1055] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:09:57 2022] GET /identity/v3/role_assignments?user.id=c83f0d0e2e8047a98fbe114ceb15c6e0&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=58422e7671624a43a2f7a18f07bb7506 => generated 259 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:57.904939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba629eb-7c48-4c0d-af10-6ff69d10019e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:57.935854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:58.084315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba629eb-7c48-4c0d-af10-6ff69d10019e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:58.124664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:58.125654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:58.125978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:58.126373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] PATH_INFO: `/v3/services/image` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:58.127447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:58.129376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:09:58.140833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:09:58.144554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:09:58.146012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 519/1056] 10.222.0.52 () {56 vars in 1105 bytes} [Sat Aug 27 02:09:57 2022] GET /identity/v3/services/image => generated 86 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:09:58.148707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:58.229247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:58.310697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916f54cc-a0dc-4c08-a984-15a7868c06f6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:58.377423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:58.378639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:58.379122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:58.379650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:58.380550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] RBAC: Adding query filter params (name=image) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:58.381093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:58.382961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:58.390277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 538/1057] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:58 2022] GET /identity/v3/services?name=image => generated 122 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:58.397583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:58.507118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:58.542727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90654d58-046a-4ddf-a47e-0ab39921a52d None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:58.606063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e55346a-6ab6-4439-a909-ec988375f782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:58.607052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55346a-6ab6-4439-a909-ec988375f782 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:58.607482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55346a-6ab6-4439-a909-ec988375f782 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:58.607922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e55346a-6ab6-4439-a909-ec988375f782 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:58.608991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e55346a-6ab6-4439-a909-ec988375f782 None admin] RBAC: Adding query filter params (type=image) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:09:58.609521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e55346a-6ab6-4439-a909-ec988375f782 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:58.612251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e55346a-6ab6-4439-a909-ec988375f782 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:58.623628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 520/1058] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:09:58 2022] GET /identity/v3/services?type=image => generated 350 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:09:58.629935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:58.828150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee65f2b4-c41d-4140-9066-498fb0dacc9d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:09:58.862396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:09:58.890030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581b64b3-085c-4187-8b02-487f93c22df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:09:58.890560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b64b3-085c-4187-8b02-487f93c22df0 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:58.890838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b64b3-085c-4187-8b02-487f93c22df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:58.891107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b64b3-085c-4187-8b02-487f93c22df0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:58.891800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581b64b3-085c-4187-8b02-487f93c22df0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:09:58.893327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581b64b3-085c-4187-8b02-487f93c22df0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:09:58.924667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 539/1059] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:09:58 2022] POST /identity/v3/endpoints => generated 327 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:09:59.351823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d86ddd-0f99-4e84-a996-375374054899 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:59.352098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d86ddd-0f99-4e84-a996-375374054899 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:59.352460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d86ddd-0f99-4e84-a996-375374054899 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:59.355232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 521/1060] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:59.361404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2de65-1252-413d-b5fd-60e78f4bf4c8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:59.362159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2de65-1252-413d-b5fd-60e78f4bf4c8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:59.362159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2de65-1252-413d-b5fd-60e78f4bf4c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:59.396203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5fb2de65-1252-413d-b5fd-60e78f4bf4c8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:59.398747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5fb2de65-1252-413d-b5fd-60e78f4bf4c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:59.507149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 540/1061] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:59.746867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555656d-8c7e-4d07-8388-3d423b6e6ffe None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:59.747562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555656d-8c7e-4d07-8388-3d423b6e6ffe None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:59.747675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555656d-8c7e-4d07-8388-3d423b6e6ffe None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:59.750120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 522/1062] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:09:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:09:59.758844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0228b5-358c-4aaa-87c7-90e9d1990811 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:09:59.759309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0228b5-358c-4aaa-87c7-90e9d1990811 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:09:59.759741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0228b5-358c-4aaa-87c7-90e9d1990811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:09:59.787828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ee0228b5-358c-4aaa-87c7-90e9d1990811 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:09:59.789716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ee0228b5-358c-4aaa-87c7-90e9d1990811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:09:59.844067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555656d-8c7e-4d07-8388-3d423b6e6ffe None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:09:59.910851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 541/1063] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:09:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:09:59.985558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555656d-8c7e-4d07-8388-3d423b6e6ffe None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:00.035172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:00.035863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:00.036222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:00.036513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:00.054623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:00.060887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:00.062826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:00.064487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 523/1064] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:09:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:00.070808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0228b5-358c-4aaa-87c7-90e9d1990811 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:00.134794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:00.251978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:00.256032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0228b5-358c-4aaa-87c7-90e9d1990811 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:00.256606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:00.307015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:00.307893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:00.308154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:00.308437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:00.309323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:00.309644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:00.311669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:00.323750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 542/1065] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:00.333369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:00.413905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47241614-1b01-47b3-862b-97c22a77c036 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:00.468450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:00.469654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:00.470077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:00.470413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:00.490611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:00.498751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:00.500428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:00.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 524/1066] 10.222.0.52 () {56 vars in 1120 bytes} [Sat Aug 27 02:10:00 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:00.507224 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:00.507453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:00.518361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e715168-df2d-4364-8154-5109071b4141 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:00.573037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:00.573781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:00.574116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:00.574400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] PATH_INFO: `/v3/users/177c0f0b47ef45789700def6963b4b1d` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:00.575757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] RBAC: Authorizing `identity:get_user(user_id=177c0f0b47ef45789700def6963b4b1d)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:00.577875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:00.585835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 543/1067] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:00 2022] GET /identity/v3/users/177c0f0b47ef45789700def6963b4b1d => generated 285 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:00.590532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:00.624297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:00.689159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c91101-e776-42b8-a03a-7d81005c0ed1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:00.750138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:00.750810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:00.750810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:00.751190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:00.751876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:00.752187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:00.753846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:00.770762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 525/1068] 10.222.0.52 () {56 vars in 1115 bytes} [Sat Aug 27 02:10:00 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:00.771591 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa35a81-931f-4e46-83ca-a7760be892ed None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:00.776323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:00.827117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:00.827835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:00.828064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:00.828371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:00.829769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] RBAC: Authorizing `identity:get_project(project_id=2e4b71fc6712404188dff9909f62cbef)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:00.831704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:00.834152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 544/1069] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:00 2022] GET /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef => generated 311 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:00.841188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:00.893863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:00.949001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf89ba31-0873-4996-b313-33824a5bc667 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:01.002329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:01.010213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2eb494-2a1b-4e69-b56e-ea7de6cab23a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:01.023883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:01.024537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.024810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.025079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:01.047771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:01.049322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:10:01.069952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:01.071959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:10:01.073916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:10:01.073916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:01.073916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:01.073916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 526/1070] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:01.077344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:01.081783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a120ea-c36a-4ab0-a8a3-4ccf31177f30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:01.082566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a120ea-c36a-4ab0-a8a3-4ccf31177f30 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.084694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a120ea-c36a-4ab0-a8a3-4ccf31177f30 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.084694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a120ea-c36a-4ab0-a8a3-4ccf31177f30 None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef/users/177c0f0b47ef45789700def6963b4b1d/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:01.090178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a120ea-c36a-4ab0-a8a3-4ccf31177f30 None admin] RBAC: Authorizing `identity:create_grant(project_id=2e4b71fc6712404188dff9909f62cbef, user_id=177c0f0b47ef45789700def6963b4b1d, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:01.092075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a120ea-c36a-4ab0-a8a3-4ccf31177f30 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:01.125165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 545/1071] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:10:00 2022] PUT /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef/users/177c0f0b47ef45789700def6963b4b1d/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:10:01.162710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431e357-9b2a-4169-92a1-f3f3f78b5933 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.163342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431e357-9b2a-4169-92a1-f3f3f78b5933 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.164032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431e357-9b2a-4169-92a1-f3f3f78b5933 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:01.166535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 546/1072] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:01 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:01.173186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777254e5-6bbb-48d7-80a0-92c49c8ebe78 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.174994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777254e5-6bbb-48d7-80a0-92c49c8ebe78 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.174994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777254e5-6bbb-48d7-80a0-92c49c8ebe78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:01.198948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-777254e5-6bbb-48d7-80a0-92c49c8ebe78 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:01.199979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-777254e5-6bbb-48d7-80a0-92c49c8ebe78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:01.284192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b02356-0735-4121-85f1-9194f34fc3b0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:01.313878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 547/1073] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:10:01.342364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:01.343094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.343573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.343815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:01.344553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:01.344973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:01.346770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:01.363741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 527/1074] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:10:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:01.379454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777254e5-6bbb-48d7-80a0-92c49c8ebe78 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:01.410188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:01.537496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777254e5-6bbb-48d7-80a0-92c49c8ebe78 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:01.589827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:01.590487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.590760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.591025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:01.601004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:01.603041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:10:01.620372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:01.623190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:10:01.624420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:01.628364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 548/1075] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:10:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:01.630177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:01.683812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:01.694239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:01.780434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3173390b-c6f5-41bc-b11f-7e43613e0ca7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:01.820515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:01.836578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:01.837235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.837525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.837809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:01.838514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:01.838816 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:01.840255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:01.862362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa4dd0a-164b-422d-8c17-4191fa005650 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:01.873343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 528/1076] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:10:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:01.882915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:01.919173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:01.919953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:01.920275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:01.920621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:02.018991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:02.029046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:02.029925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf53fc7-829a-4e2d-9d52-7f339c9d11a1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:02.030857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:02.097574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:02.098693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:02.099206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:02.099661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:02.116857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:02.119379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:10:02.143180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:02.145159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:10:02.146953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:10:02.146953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:02.146953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:02.149634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 529/1077] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:02.152286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:02.174959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:02.210978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 549/1078] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 528 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 27 02:10:02.219219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:02.318810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b1d69d-6b28-4a5c-b949-d0c610d4d579 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:02.384730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:02.385279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:02.387322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:02.387776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:02.388587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:02.389057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:02.391721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:02.412578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93777060-5afa-48c2-85f1-d3a5e942669b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:02.413578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 530/1079] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:02.418370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:02.438284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:02.464512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:02.465729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:02.466344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:02.466911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:02.488757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:02.490705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:10:02.518458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:02.519904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:10:02.522898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:10:02.522898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:02.522898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:02.522898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 550/1080] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:02.528183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:02.586275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:02.600455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0945f7-b9c4-4a8d-b72a-7cfea95d94af None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:02.644246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:02.644885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:02.645305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:02.645610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:02.657984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:02.658935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:10:02.672362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:02.674987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:02.679596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 531/1081] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:10:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:02.682125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:02.700920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3148d-5004-4f56-b1e6-2500c134f06f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:02.756448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:02.757219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:02.757567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:02.757914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:02.758815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:02.759118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:02.760932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:02.778298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 551/1082] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:02.791476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:02.868898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79759f2-df92-4140-b32c-d53e9474b985 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:02.934363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c474343-5d76-430b-9cde-acf791088ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:02.935226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c474343-5d76-430b-9cde-acf791088ee8 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:02.935676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c474343-5d76-430b-9cde-acf791088ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:02.936119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c474343-5d76-430b-9cde-acf791088ee8 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:02.937393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c474343-5d76-430b-9cde-acf791088ee8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:02.937899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c474343-5d76-430b-9cde-acf791088ee8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:02.941285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c474343-5d76-430b-9cde-acf791088ee8 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:02.964053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 532/1083] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:10:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:02.972999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:03.128501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:03.140231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e51e17-d383-4d29-975e-a7ff0aea8d46 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:03.199390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fd1a9d-3237-4560-9b14-eea773e20dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:03.200384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fd1a9d-3237-4560-9b14-eea773e20dc7 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:03.200828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fd1a9d-3237-4560-9b14-eea773e20dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:03.201380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fd1a9d-3237-4560-9b14-eea773e20dc7 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/c83f0d0e2e8047a98fbe114ceb15c6e0/roles/58422e7671624a43a2f7a18f07bb7506` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:03.204961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fd1a9d-3237-4560-9b14-eea773e20dc7 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=c83f0d0e2e8047a98fbe114ceb15c6e0, role_id=58422e7671624a43a2f7a18f07bb7506)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:03.206809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fd1a9d-3237-4560-9b14-eea773e20dc7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:03.228127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 552/1084] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:10:02 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/c83f0d0e2e8047a98fbe114ceb15c6e0/roles/58422e7671624a43a2f7a18f07bb7506 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:10:03.480727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991b363-ae37-4924-adac-65b6ed12eabf None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:03.480986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991b363-ae37-4924-adac-65b6ed12eabf None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:03.481315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991b363-ae37-4924-adac-65b6ed12eabf None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:03.482842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 533/1085] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:03.491714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eb5176-927c-46b6-a4ee-e326a17eaede None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:03.492149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eb5176-927c-46b6-a4ee-e326a17eaede None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:03.492549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eb5176-927c-46b6-a4ee-e326a17eaede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:03.517144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-24eb5176-927c-46b6-a4ee-e326a17eaede None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:03.518631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-24eb5176-927c-46b6-a4ee-e326a17eaede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:03.606626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 553/1086] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:03.885270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991b363-ae37-4924-adac-65b6ed12eabf None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:04.007828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991b363-ae37-4924-adac-65b6ed12eabf None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:04.045900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:04.055492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:04.056142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:04.056411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:04.056695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:04.071136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:04.076050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:04.077096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:04.078754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 534/1087] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:10:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:04.083962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eb5176-927c-46b6-a4ee-e326a17eaede None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:04.184342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:04.207170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eb5176-927c-46b6-a4ee-e326a17eaede None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:04.249292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:04.250113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:04.250458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:04.250726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:04.251402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:04.251702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:04.254328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:04.263845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 554/1088] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:04.270844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:04.387293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0250c2c2-80c1-4199-894a-243a70d307f0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:04.426705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:04.427330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:04.427611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:04.427879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] PATH_INFO: `/v3/users/177c0f0b47ef45789700def6963b4b1d` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:04.429023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] RBAC: Authorizing `identity:get_user(user_id=177c0f0b47ef45789700def6963b4b1d)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:04.430490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:04.432101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 535/1089] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:04 2022] GET /identity/v3/users/177c0f0b47ef45789700def6963b4b1d => generated 285 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:04.438526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:04.543653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07edf96-0482-4611-8466-1be1277a30bb None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:04.579195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:04.579796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:04.580294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:04.580490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:04.599030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:04.667105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:04.668841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:04.789105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 555/1090] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:04.790434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:04.854070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:04.903132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5f8870-552d-4849-855c-1f499632fb1b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:04.939949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a539e-1c4e-4c18-941f-a3249629093a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:04.940627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a539e-1c4e-4c18-941f-a3249629093a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:04.940893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a539e-1c4e-4c18-941f-a3249629093a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:04.941207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a539e-1c4e-4c18-941f-a3249629093a None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:04.942421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a539e-1c4e-4c18-941f-a3249629093a None admin] RBAC: Authorizing `identity:get_project(project_id=2e4b71fc6712404188dff9909f62cbef)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:04.944320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a539e-1c4e-4c18-941f-a3249629093a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:04.946232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 536/1091] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:04 2022] GET /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef => generated 311 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:04.953090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:05.004316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:05.078616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f7766e-26c8-48c9-b51a-38136a5bd8ad None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:05.127532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be91259f-5f35-4bf4-9db3-3130958aa3a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:05.127532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be91259f-5f35-4bf4-9db3-3130958aa3a9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:05.128256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be91259f-5f35-4bf4-9db3-3130958aa3a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:05.128427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be91259f-5f35-4bf4-9db3-3130958aa3a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:05.129199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be91259f-5f35-4bf4-9db3-3130958aa3a9 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=2e4b71fc6712404188dff9909f62cbef, user.id=177c0f0b47ef45789700def6963b4b1d) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:05.129531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be91259f-5f35-4bf4-9db3-3130958aa3a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:05.130979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be91259f-5f35-4bf4-9db3-3130958aa3a9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:05.140515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 556/1092] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:04 2022] GET /identity/v3/role_assignments?user.id=177c0f0b47ef45789700def6963b4b1d&scope.project.id=2e4b71fc6712404188dff9909f62cbef&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:05.232810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f82673-790c-4826-9965-8d840aa63ebd None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:05.233083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f82673-790c-4826-9965-8d840aa63ebd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:05.233412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f82673-790c-4826-9965-8d840aa63ebd None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:05.234855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 537/1093] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:05.242165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ac702-8aa2-4793-9bdc-47670a0825c7 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:05.242587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ac702-8aa2-4793-9bdc-47670a0825c7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:05.242756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ac702-8aa2-4793-9bdc-47670a0825c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:05.263674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-819ac702-8aa2-4793-9bdc-47670a0825c7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:05.265375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-819ac702-8aa2-4793-9bdc-47670a0825c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:05.357034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 557/1094] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:05.664085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f82673-790c-4826-9965-8d840aa63ebd None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:05.787851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f82673-790c-4826-9965-8d840aa63ebd None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:05.816146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:05.830306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:05.831390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:05.831725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:05.832044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:05.845199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:05.850639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:05.851780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 27 02:10:05.852705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:05.857397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 538/1095] 10.222.0.52 () {56 vars in 1120 bytes} [Sat Aug 27 02:10:05 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:05.858457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ac702-8aa2-4793-9bdc-47670a0825c7 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:05.909277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:05.978732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ac702-8aa2-4793-9bdc-47670a0825c7 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:06.014663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:06.015239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:06.015514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:06.015842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:06.016425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:06.016725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:06.018451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:06.026955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 558/1096] 10.222.0.52 () {56 vars in 1115 bytes} [Sat Aug 27 02:10:05 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:06.033469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:06.152278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67af7f73-9b24-4a5c-a894-7198e2ad409f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:06.204451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:06.205492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:06.205976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:06.206451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:06.220341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:06.222187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:10:06.237445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:06.238832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:10:06.240066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:10:06.240066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:06.240066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:06.240066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 539/1097] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:06.245577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:06.360594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:06.364196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a1b49d-8f3e-46e5-94fb-a2605ec3e207 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:06.400738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:06.401372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:06.401645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:06.401913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:06.402561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:06.402861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:06.404222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:06.417971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 559/1098] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:10:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:06.423928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:06.463017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:06.539761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329c1bc7-f810-4815-ba16-692e9afb1b13 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:06.590678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:06.591576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:06.591993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:06.592391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:06.604207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:06.606081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:10:06.621269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:06.623108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:06.627185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 540/1099] 10.222.0.52 () {56 vars in 1093 bytes} [Sat Aug 27 02:10:06 2022] GET /identity/v3/users/nova => generated 82 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:06.629459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:06.746157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b9b2f-e72a-4c66-bbdf-01e7c9b72f2c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:06.786777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdaea17-6923-4f66-9572-d14d15e56e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:06.787399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdaea17-6923-4f66-9572-d14d15e56e8a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:06.787674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdaea17-6923-4f66-9572-d14d15e56e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:06.787941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdaea17-6923-4f66-9572-d14d15e56e8a None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:06.788680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdaea17-6923-4f66-9572-d14d15e56e8a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:06.788984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdaea17-6923-4f66-9572-d14d15e56e8a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:06.790515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdaea17-6923-4f66-9572-d14d15e56e8a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:06.811548 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 560/1100] 10.222.0.52 () {56 vars in 1133 bytes} [Sat Aug 27 02:10:06 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:06.818205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:06.941624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:06.950810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428a2cc-0a22-4c39-9e13-01bf19285e43 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:06.990517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:06.991226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:06.991500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:06.991767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.076459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:07.078230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:07.160636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c821fed-3b66-4c87-8c7f-caa719fd0602 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.161418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c821fed-3b66-4c87-8c7f-caa719fd0602 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.161418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c821fed-3b66-4c87-8c7f-caa719fd0602 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.162736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 561/1101] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:10:07.168321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e874fa-d369-43a0-a74e-295be09605b5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.168601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e874fa-d369-43a0-a74e-295be09605b5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.168880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e874fa-d369-43a0-a74e-295be09605b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.187133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-54e874fa-d369-43a0-a74e-295be09605b5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:07.188348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-54e874fa-d369-43a0-a74e-295be09605b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:07.190201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:07.206691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 541/1102] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:07.206941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:07.208646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:07.289568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 562/1103] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:10:07.347805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4b198-ef0e-422b-bb7f-1bbdad1f53d7 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:07.386047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:07.386674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.386946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.387208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.398753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:07.400339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:10:07.415939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:07.417077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:10:07.418002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:10:07.418002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:10:07.418002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:07.418002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 542/1104] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:07.422928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e874fa-d369-43a0-a74e-295be09605b5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:07.536104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e874fa-d369-43a0-a74e-295be09605b5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:07.569236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:07.583590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:07.584209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.584485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.584758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.585436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:07.585738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:07.587163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:07.599736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 563/1105] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:10:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:07.605547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:07.618010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:07.629421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:07.693604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aee3a6-53fc-45ab-a220-3ac6ef603ea5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:07.732512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:07.733165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.733438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.733705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.734799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:07.736254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:07.738501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 543/1106] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:07.741852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec213f5-14da-4df4-a949-a5754bf14a08 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:07.744952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:07.783708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:07.784549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.784833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.785092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.792775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:07.794297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:10:07.806734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:07.807836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:07.809206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 564/1107] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:10:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:07.813975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:07.863792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b193d0-a086-4c41-93e4-4f59f979814b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:07.882025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:07.888122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:07.906603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:07.907459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.907814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.908151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.909315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:07.911189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:07.930412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edc45a2-cd1e-4953-91a1-adf2d1839a7c None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:07.940717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 544/1108] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:10:07 2022] POST /identity/v3/users => generated 286 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:10:07.974640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c081dd05-d909-4d4c-a27b-b7f501e037c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:07.975274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c081dd05-d909-4d4c-a27b-b7f501e037c7 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:07.975552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c081dd05-d909-4d4c-a27b-b7f501e037c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:07.975911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c081dd05-d909-4d4c-a27b-b7f501e037c7 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:07.976671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c081dd05-d909-4d4c-a27b-b7f501e037c7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:07.977101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c081dd05-d909-4d4c-a27b-b7f501e037c7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:07.978982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c081dd05-d909-4d4c-a27b-b7f501e037c7 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:07.997359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 565/1109] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:10:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:08.005469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:08.137766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327647e4-f1c2-4142-89e3-5c3375cc3b9d None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:08.184748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bc3e71-5363-4ae8-80ce-2f2d131dc056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:08.185461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bc3e71-5363-4ae8-80ce-2f2d131dc056 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:08.185742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bc3e71-5363-4ae8-80ce-2f2d131dc056 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:08.186013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bc3e71-5363-4ae8-80ce-2f2d131dc056 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:08.186766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bc3e71-5363-4ae8-80ce-2f2d131dc056 None admin] RBAC: Adding query filter params (role.id=58422e7671624a43a2f7a18f07bb7506, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=c83f0d0e2e8047a98fbe114ceb15c6e0) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:08.187075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bc3e71-5363-4ae8-80ce-2f2d131dc056 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:08.188510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bc3e71-5363-4ae8-80ce-2f2d131dc056 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:08.196797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 545/1110] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:08 2022] GET /identity/v3/role_assignments?user.id=c83f0d0e2e8047a98fbe114ceb15c6e0&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=58422e7671624a43a2f7a18f07bb7506 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:09.953458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d76ebb4-aa73-4f8c-aab5-148f00393be7 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:09.953997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d76ebb4-aa73-4f8c-aab5-148f00393be7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:09.953997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d76ebb4-aa73-4f8c-aab5-148f00393be7 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:09.955331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 566/1111] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:09.960924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c92c34-e1bb-4d5f-9679-569fa2f7bd60 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:09.961410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c92c34-e1bb-4d5f-9679-569fa2f7bd60 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:09.961499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c92c34-e1bb-4d5f-9679-569fa2f7bd60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:09.980979 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-03c92c34-e1bb-4d5f-9679-569fa2f7bd60 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:09.982166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-03c92c34-e1bb-4d5f-9679-569fa2f7bd60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:10.015359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:10.061467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 546/1112] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:10:10.327676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d76ebb4-aa73-4f8c-aab5-148f00393be7 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:10.441728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d76ebb4-aa73-4f8c-aab5-148f00393be7 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:10.469444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:10.478580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:10.479217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:10.479494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:10.479767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] PATH_INFO: `/v3/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:10.481225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] RBAC: Authorizing `identity:get_role(role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:10.482613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:10.484053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 567/1113] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:10 2022] GET /identity/v3/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 223 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:10.490458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c92c34-e1bb-4d5f-9679-569fa2f7bd60 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:10.599783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c92c34-e1bb-4d5f-9679-569fa2f7bd60 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:10.633724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:10.634350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:10.634620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:10.634886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] PATH_INFO: `/v3/users/c6fc2b2c0d6342aebb32052097e87c18` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:10.649193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] RBAC: Authorizing `identity:get_user(user_id=c6fc2b2c0d6342aebb32052097e87c18)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:10.650642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:10.652319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 547/1114] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:10 2022] GET /identity/v3/users/c6fc2b2c0d6342aebb32052097e87c18 => generated 286 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:10.658375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:10.777351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f44a49-abc9-49f2-8d53-32fb1e28af1e None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:10.817407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:10.818018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:10.818294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:10.818572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:10.901083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:10.909057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:10.910639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:11.016763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 568/1115] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:10 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:11.023572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:11.061257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:11.153403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135702ca-128e-4c67-95e2-f80059d786d6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:11.204752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26520bb1-2702-4491-9321-dc28480487ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:11.205860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26520bb1-2702-4491-9321-dc28480487ff None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:11.206282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26520bb1-2702-4491-9321-dc28480487ff None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:11.206728 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26520bb1-2702-4491-9321-dc28480487ff None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:11.208431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26520bb1-2702-4491-9321-dc28480487ff None admin] RBAC: Authorizing `identity:get_project(project_id=2e4b71fc6712404188dff9909f62cbef)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:11.210782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26520bb1-2702-4491-9321-dc28480487ff None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:11.213438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 548/1116] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:11 2022] GET /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef => generated 311 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:11.220132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:11.331003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:11.339222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b9d4e-3462-4161-9d4f-9d22644b0d42 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:11.390579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42225733-bd28-45b7-abdf-e8de8023f4e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:11.391251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42225733-bd28-45b7-abdf-e8de8023f4e2 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:11.391561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42225733-bd28-45b7-abdf-e8de8023f4e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:11.391859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42225733-bd28-45b7-abdf-e8de8023f4e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:11.392822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42225733-bd28-45b7-abdf-e8de8023f4e2 None admin] RBAC: Adding query filter params (role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=2e4b71fc6712404188dff9909f62cbef, user.id=c6fc2b2c0d6342aebb32052097e87c18) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:11.393300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42225733-bd28-45b7-abdf-e8de8023f4e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:11.394755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42225733-bd28-45b7-abdf-e8de8023f4e2 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:11.401012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 569/1117] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:11 2022] GET /identity/v3/role_assignments?user.id=c6fc2b2c0d6342aebb32052097e87c18&scope.project.id=2e4b71fc6712404188dff9909f62cbef&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:13.307637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71741851-05f7-4394-a15e-90e71e2aedec None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:13.308275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71741851-05f7-4394-a15e-90e71e2aedec None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:13.308275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71741851-05f7-4394-a15e-90e71e2aedec None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:13.309523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 549/1118] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:10:13.315428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ca228-dcf3-4555-b59e-8fb915df0524 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:13.315969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ca228-dcf3-4555-b59e-8fb915df0524 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:13.316066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ca228-dcf3-4555-b59e-8fb915df0524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:13.340503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3b4ca228-dcf3-4555-b59e-8fb915df0524 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:13.342378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3b4ca228-dcf3-4555-b59e-8fb915df0524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:13.417283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 570/1119] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:13.695301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71741851-05f7-4394-a15e-90e71e2aedec None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:13.798181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:13.809850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71741851-05f7-4394-a15e-90e71e2aedec None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:13.846480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:13.847171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:13.847444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:13.847701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] PATH_INFO: `/v3/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:13.849169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] RBAC: Authorizing `identity:get_role(role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:13.850514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:13.851958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 550/1120] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:13 2022] GET /identity/v3/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 223 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:13.860424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4ca228-dcf3-4555-b59e-8fb915df0524 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:13.964050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:13.976860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4ca228-dcf3-4555-b59e-8fb915df0524 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:14.014699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:14.015341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:14.015618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:14.015887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] PATH_INFO: `/v3/users/c6fc2b2c0d6342aebb32052097e87c18` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:14.017077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] RBAC: Authorizing `identity:get_user(user_id=c6fc2b2c0d6342aebb32052097e87c18)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:14.018601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:14.020309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 571/1121] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:13 2022] GET /identity/v3/users/c6fc2b2c0d6342aebb32052097e87c18 => generated 286 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:14.026572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:14.134558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e5e9f0-3ee9-49e1-9939-620cff6104cf None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:14.175514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1030052-fc3f-4cf0-8e4d-17f6ed9a1e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:14.176158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1030052-fc3f-4cf0-8e4d-17f6ed9a1e54 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:14.176410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1030052-fc3f-4cf0-8e4d-17f6ed9a1e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:14.176685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1030052-fc3f-4cf0-8e4d-17f6ed9a1e54 None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:14.177883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1030052-fc3f-4cf0-8e4d-17f6ed9a1e54 None admin] RBAC: Authorizing `identity:get_project(project_id=2e4b71fc6712404188dff9909f62cbef)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:14.179308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1030052-fc3f-4cf0-8e4d-17f6ed9a1e54 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:14.180970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 551/1122] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:14 2022] GET /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:14.187296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:14.306121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdee4aa-60cf-40a5-bcb2-f3316125dd62 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:14.344031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d61ab7-1de3-48b0-82fe-adc27d69ff7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:14.344746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d61ab7-1de3-48b0-82fe-adc27d69ff7d None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:14.345030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d61ab7-1de3-48b0-82fe-adc27d69ff7d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:14.345369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d61ab7-1de3-48b0-82fe-adc27d69ff7d None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef/users/c6fc2b2c0d6342aebb32052097e87c18/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:14.347856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d61ab7-1de3-48b0-82fe-adc27d69ff7d None admin] RBAC: Authorizing `identity:create_grant(project_id=2e4b71fc6712404188dff9909f62cbef, user_id=c6fc2b2c0d6342aebb32052097e87c18, role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:14.349313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d61ab7-1de3-48b0-82fe-adc27d69ff7d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:14.363612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 572/1123] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:10:14 2022] PUT /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef/users/c6fc2b2c0d6342aebb32052097e87c18/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:10:16.261977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f780a2-a607-415e-98d4-ebb5e16e391a None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:16.262703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f780a2-a607-415e-98d4-ebb5e16e391a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:16.262703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f780a2-a607-415e-98d4-ebb5e16e391a None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:16.263853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 552/1124] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:16.269448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f271dc-ea68-4e39-9c76-0ab42218293c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:16.269997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f271dc-ea68-4e39-9c76-0ab42218293c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:16.269997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f271dc-ea68-4e39-9c76-0ab42218293c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:16.288149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-55f271dc-ea68-4e39-9c76-0ab42218293c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:16.289359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-55f271dc-ea68-4e39-9c76-0ab42218293c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:16.302789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:16.362411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 573/1125] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:10:16.633645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f780a2-a607-415e-98d4-ebb5e16e391a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:16.662937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:16.740891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f780a2-a607-415e-98d4-ebb5e16e391a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:16.776248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:16.776869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:16.777168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:16.777784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] PATH_INFO: `/v3/roles/d6f4d629ec204914895b2e6145cd2f01` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:16.779009 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] RBAC: Authorizing `identity:get_role(role_id=d6f4d629ec204914895b2e6145cd2f01)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:16.780354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:16.781735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 553/1126] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:16 2022] GET /identity/v3/roles/d6f4d629ec204914895b2e6145cd2f01 => generated 223 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:16.788238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f271dc-ea68-4e39-9c76-0ab42218293c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:16.891202 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f271dc-ea68-4e39-9c76-0ab42218293c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:16.926078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:16.926730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:16.927001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:16.927266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] PATH_INFO: `/v3/users/c6fc2b2c0d6342aebb32052097e87c18` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:16.928392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] RBAC: Authorizing `identity:get_user(user_id=c6fc2b2c0d6342aebb32052097e87c18)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:16.929851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:16.931452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 574/1127] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:16 2022] GET /identity/v3/users/c6fc2b2c0d6342aebb32052097e87c18 => generated 286 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:16.937674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:17.040696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:17.047288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c6ff6-6415-45ed-ad35-9b290f5b110d None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:17.082589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:17.083197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:17.083471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:17.083740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:17.151787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:17.153176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:17.246799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:17.256202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 554/1128] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:17.262575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:17.282417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:17.367131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00765469-65a7-4e13-b8af-8be15321c941 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:17.409037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24375664-ad4c-46e0-a762-76593132b264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:17.409751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24375664-ad4c-46e0-a762-76593132b264 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:17.409861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24375664-ad4c-46e0-a762-76593132b264 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:17.410157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24375664-ad4c-46e0-a762-76593132b264 None admin] PATH_INFO: `/v3/projects/2e4b71fc6712404188dff9909f62cbef` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:17.411317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24375664-ad4c-46e0-a762-76593132b264 None admin] RBAC: Authorizing `identity:get_project(project_id=2e4b71fc6712404188dff9909f62cbef)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:17.412696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24375664-ad4c-46e0-a762-76593132b264 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:17.414410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 575/1129] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:17 2022] GET /identity/v3/projects/2e4b71fc6712404188dff9909f62cbef => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:17.421073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:17.524146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cef558-32ec-4bf8-9e54-36bc2139ef19 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:17.559038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d328a3ae-808e-477c-b7ca-f18ac9c21657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:17.559634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d328a3ae-808e-477c-b7ca-f18ac9c21657 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:17.559914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d328a3ae-808e-477c-b7ca-f18ac9c21657 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:17.560176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d328a3ae-808e-477c-b7ca-f18ac9c21657 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:17.560912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d328a3ae-808e-477c-b7ca-f18ac9c21657 None admin] RBAC: Adding query filter params (role.id=d6f4d629ec204914895b2e6145cd2f01, scope.project.id=2e4b71fc6712404188dff9909f62cbef, user.id=c6fc2b2c0d6342aebb32052097e87c18) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:17.561250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d328a3ae-808e-477c-b7ca-f18ac9c21657 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:17.562674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d328a3ae-808e-477c-b7ca-f18ac9c21657 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:17.568430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 555/1130] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:17 2022] GET /identity/v3/role_assignments?user.id=c6fc2b2c0d6342aebb32052097e87c18&scope.project.id=2e4b71fc6712404188dff9909f62cbef&role.id=d6f4d629ec204914895b2e6145cd2f01 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:19.510497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4aabb8-a6c4-49b6-82c9-4c95b46cd62e None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:19.511083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4aabb8-a6c4-49b6-82c9-4c95b46cd62e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:19.511083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4aabb8-a6c4-49b6-82c9-4c95b46cd62e None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:19.512565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 576/1131] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:19.518163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94045926-4352-40f4-9c25-5a78868e1f67 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:19.518633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94045926-4352-40f4-9c25-5a78868e1f67 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:19.518741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94045926-4352-40f4-9c25-5a78868e1f67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:19.537001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-94045926-4352-40f4-9c25-5a78868e1f67 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:19.538218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-94045926-4352-40f4-9c25-5a78868e1f67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:19.593446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:19.606653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 556/1132] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:10:19.877545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4aabb8-a6c4-49b6-82c9-4c95b46cd62e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:19.968952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:19.986411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4aabb8-a6c4-49b6-82c9-4c95b46cd62e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:20.021659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9e86e9-1767-46fe-b7d4-264006b99084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:20.022280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e86e9-1767-46fe-b7d4-264006b99084 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:20.022555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e86e9-1767-46fe-b7d4-264006b99084 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:20.022827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e86e9-1767-46fe-b7d4-264006b99084 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:20.023938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9e86e9-1767-46fe-b7d4-264006b99084 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:20.025394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9e86e9-1767-46fe-b7d4-264006b99084 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:20.026996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 577/1133] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:20.033011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94045926-4352-40f4-9c25-5a78868e1f67 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:20.160384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94045926-4352-40f4-9c25-5a78868e1f67 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:20.211789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a71ae8-9519-4631-8286-998e9d839511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:20.212684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a71ae8-9519-4631-8286-998e9d839511 None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:20.213086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a71ae8-9519-4631-8286-998e9d839511 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:20.213572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a71ae8-9519-4631-8286-998e9d839511 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:20.214609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a71ae8-9519-4631-8286-998e9d839511 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:20.216975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a71ae8-9519-4631-8286-998e9d839511 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:20.252855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 557/1134] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:10:20 2022] POST /identity/v3/projects => generated 311 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:10:22.234611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c81a1-7d02-4178-a4c7-525a8ad2ceeb None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:22.235176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c81a1-7d02-4178-a4c7-525a8ad2ceeb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:22.235176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c81a1-7d02-4178-a4c7-525a8ad2ceeb None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:22.236526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 578/1135] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:22.242031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9605697f-d80e-488e-b859-1d366c841684 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:22.242415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9605697f-d80e-488e-b859-1d366c841684 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:22.242579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9605697f-d80e-488e-b859-1d366c841684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:22.262228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9605697f-d80e-488e-b859-1d366c841684 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:22.263433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9605697f-d80e-488e-b859-1d366c841684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:22.301913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:22.342435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 558/1136] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:22.616442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20c81a1-7d02-4178-a4c7-525a8ad2ceeb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:22.744268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20c81a1-7d02-4178-a4c7-525a8ad2ceeb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:22.781946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f8d223-5303-419e-847d-66b37b80766b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:22.782624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8d223-5303-419e-847d-66b37b80766b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:22.782902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8d223-5303-419e-847d-66b37b80766b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:22.783210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f8d223-5303-419e-847d-66b37b80766b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:22.784558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f8d223-5303-419e-847d-66b37b80766b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:22.786453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f8d223-5303-419e-847d-66b37b80766b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:22.788245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 579/1137] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:10:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:22.794813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9605697f-d80e-488e-b859-1d366c841684 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:22.904447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9605697f-d80e-488e-b859-1d366c841684 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:22.943155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b2f188-8325-4ca3-9c9b-53ae9bf395bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:22.943767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b2f188-8325-4ca3-9c9b-53ae9bf395bd None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:22.944033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b2f188-8325-4ca3-9c9b-53ae9bf395bd None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:22.944298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b2f188-8325-4ca3-9c9b-53ae9bf395bd None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:22.945297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b2f188-8325-4ca3-9c9b-53ae9bf395bd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:22.946697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b2f188-8325-4ca3-9c9b-53ae9bf395bd None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:22.969233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 559/1138] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:10:22 2022] POST /identity/v3/users => generated 286 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:10:24.962160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f046930-d0a6-4c63-afa9-4265f5a36a66 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:24.962716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f046930-d0a6-4c63-afa9-4265f5a36a66 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:24.962716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f046930-d0a6-4c63-afa9-4265f5a36a66 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:24.964046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 580/1139] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:24.969791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c82828-1ceb-44f7-b534-0eaaf25d4b15 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:24.970228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c82828-1ceb-44f7-b534-0eaaf25d4b15 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:24.970327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c82828-1ceb-44f7-b534-0eaaf25d4b15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:24.991098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-94c82828-1ceb-44f7-b534-0eaaf25d4b15 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:24.992307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-94c82828-1ceb-44f7-b534-0eaaf25d4b15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:25.018033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:25.068872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 560/1140] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:25.350155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f046930-d0a6-4c63-afa9-4265f5a36a66 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:25.376006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:25.468259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f046930-d0a6-4c63-afa9-4265f5a36a66 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:25.512264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:25.513186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:25.513629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:25.514041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:25.529352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:25.537520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:25.538743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:25.539882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 581/1141] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:10:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:25.545662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c82828-1ceb-44f7-b534-0eaaf25d4b15 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:25.660091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c82828-1ceb-44f7-b534-0eaaf25d4b15 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:25.699803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:25.700404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:25.700673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:25.700936 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:25.701644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:25.701942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:25.703325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:25.712867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 561/1142] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:25.719342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:25.815082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:25.836647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e253a207-357c-4db9-952c-8040e9e5db47 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:25.878631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:25.879425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:25.879697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:25.879964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] PATH_INFO: `/v3/users/83a6816468fd49d8a75f0f090f56da95` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:25.895997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] RBAC: Authorizing `identity:get_user(user_id=83a6816468fd49d8a75f0f090f56da95)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:25.897669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:25.899528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 582/1143] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:25 2022] GET /identity/v3/users/83a6816468fd49d8a75f0f090f56da95 => generated 286 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:25.906099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:25.931609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:26.022088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3c53d0-6635-4474-aac8-fce3cc51b9ea None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:26.058791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:26.059391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:26.059666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:26.059928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:26.139606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:26.141164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:26.163744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:26.244442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 562/1144] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:25 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:26.251024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:26.358319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b56e4-abff-4e34-a8f3-94d371795cd3 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:26.402714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21b3678-def8-4055-89bb-d0127f96022e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:26.403364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b3678-def8-4055-89bb-d0127f96022e None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:26.403623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b3678-def8-4055-89bb-d0127f96022e None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:26.403881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b3678-def8-4055-89bb-d0127f96022e None admin] PATH_INFO: `/v3/projects/c88b3d709c9b437d8775547503907a45` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:26.405054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21b3678-def8-4055-89bb-d0127f96022e None admin] RBAC: Authorizing `identity:get_project(project_id=c88b3d709c9b437d8775547503907a45)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:26.406507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21b3678-def8-4055-89bb-d0127f96022e None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:26.408131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 583/1145] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:26 2022] GET /identity/v3/projects/c88b3d709c9b437d8775547503907a45 => generated 311 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:26.414927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:26.533642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eed39ac-ec34-476f-9055-3d97a51f6c02 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:26.561149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:26.577905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f2074e-57d9-4058-8225-63ddd460e33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:26.578621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2074e-57d9-4058-8225-63ddd460e33c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:26.578895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2074e-57d9-4058-8225-63ddd460e33c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:26.579161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2074e-57d9-4058-8225-63ddd460e33c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:26.579894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f2074e-57d9-4058-8225-63ddd460e33c None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=c88b3d709c9b437d8775547503907a45, user.id=83a6816468fd49d8a75f0f090f56da95) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:26.580193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f2074e-57d9-4058-8225-63ddd460e33c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:26.581678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f2074e-57d9-4058-8225-63ddd460e33c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:26.587299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 563/1146] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:26 2022] GET /identity/v3/role_assignments?user.id=83a6816468fd49d8a75f0f090f56da95&scope.project.id=c88b3d709c9b437d8775547503907a45&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:28.485270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79d8684-a207-4dd8-986e-b054195283e7 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:28.485849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79d8684-a207-4dd8-986e-b054195283e7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:28.485849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79d8684-a207-4dd8-986e-b054195283e7 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:28.487132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 584/1147] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:28.493445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace41f4-748c-42cc-b96a-68f0167d6f35 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:28.493924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace41f4-748c-42cc-b96a-68f0167d6f35 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:28.494014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace41f4-748c-42cc-b96a-68f0167d6f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:28.512491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-bace41f4-748c-42cc-b96a-68f0167d6f35 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:28.513827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bace41f4-748c-42cc-b96a-68f0167d6f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:28.580589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 564/1148] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:10:28.856152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79d8684-a207-4dd8-986e-b054195283e7 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:28.865627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:28.960499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79d8684-a207-4dd8-986e-b054195283e7 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:28.995248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:28.995887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:28.996164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:28.996434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:29.008002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:29.012810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:29.013984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:29.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 585/1149] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:10:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:29.020725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace41f4-748c-42cc-b96a-68f0167d6f35 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:29.122164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace41f4-748c-42cc-b96a-68f0167d6f35 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:29.136909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:29.159086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:29.159767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:29.160110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:29.160389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:29.161931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:29.161931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:29.163211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:29.173917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 565/1150] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:29.180009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:29.256439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:29.309101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788d39f3-471f-442a-bd1c-a79f3ac0abdc None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:29.347006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:29.347721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:29.348173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:29.348420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] PATH_INFO: `/v3/users/83a6816468fd49d8a75f0f090f56da95` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:29.349724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] RBAC: Authorizing `identity:get_user(user_id=83a6816468fd49d8a75f0f090f56da95)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:29.351156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:29.352728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 586/1151] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:29 2022] GET /identity/v3/users/83a6816468fd49d8a75f0f090f56da95 => generated 286 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:29.358815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:29.466863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8aa22-8c69-4df0-b0b2-e06f5a74c6a0 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:29.504314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898e2c4c-6b94-446d-a9e6-5868d26c6203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:29.505108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e2c4c-6b94-446d-a9e6-5868d26c6203 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:29.505234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e2c4c-6b94-446d-a9e6-5868d26c6203 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:29.505479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e2c4c-6b94-446d-a9e6-5868d26c6203 None admin] PATH_INFO: `/v3/projects/c88b3d709c9b437d8775547503907a45` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:29.506985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898e2c4c-6b94-446d-a9e6-5868d26c6203 None admin] RBAC: Authorizing `identity:get_project(project_id=c88b3d709c9b437d8775547503907a45)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:29.509084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898e2c4c-6b94-446d-a9e6-5868d26c6203 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:29.510772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 566/1152] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:29 2022] GET /identity/v3/projects/c88b3d709c9b437d8775547503907a45 => generated 311 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:29.517660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:29.623918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bf6fbd-525b-44dc-bd65-d61dee34a5c5 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:29.658623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c821a036-b581-4692-8602-5448f59cce21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:29.659276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821a036-b581-4692-8602-5448f59cce21 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:29.659547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821a036-b581-4692-8602-5448f59cce21 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:29.660480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821a036-b581-4692-8602-5448f59cce21 None admin] PATH_INFO: `/v3/projects/c88b3d709c9b437d8775547503907a45/users/83a6816468fd49d8a75f0f090f56da95/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:29.661841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c821a036-b581-4692-8602-5448f59cce21 None admin] RBAC: Authorizing `identity:create_grant(project_id=c88b3d709c9b437d8775547503907a45, user_id=83a6816468fd49d8a75f0f090f56da95, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:29.663275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c821a036-b581-4692-8602-5448f59cce21 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:29.665603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:29.679090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 587/1153] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:10:29 2022] PUT /identity/v3/projects/c88b3d709c9b437d8775547503907a45/users/83a6816468fd49d8a75f0f090f56da95/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:10:31.562312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6613e2-73c7-4de0-adab-bafe8840f51e None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:31.562927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6613e2-73c7-4de0-adab-bafe8840f51e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:31.562927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6613e2-73c7-4de0-adab-bafe8840f51e None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:31.564239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 567/1154] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:31.570218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ef68b-f56f-4475-871e-6d88fb1b5873 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:31.570683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ef68b-f56f-4475-871e-6d88fb1b5873 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:31.571099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ef68b-f56f-4475-871e-6d88fb1b5873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:31.589358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e82ef68b-f56f-4475-871e-6d88fb1b5873 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:31.590547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e82ef68b-f56f-4475-871e-6d88fb1b5873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:31.657437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 588/1155] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:31.929152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6613e2-73c7-4de0-adab-bafe8840f51e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:31.978918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:32.037348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6613e2-73c7-4de0-adab-bafe8840f51e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:32.074436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:32.075059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:32.075329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:32.075563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:32.086512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:32.091257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:32.092361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:32.093821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 568/1156] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:10:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:32.099359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ef68b-f56f-4475-871e-6d88fb1b5873 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:32.199321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ef68b-f56f-4475-871e-6d88fb1b5873 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:32.224242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:32.235989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:32.236601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:32.236873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:32.237106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:32.237830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:32.238168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:32.239640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:32.248214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 589/1157] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:32.254575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:32.360313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3a0ce-dcc7-4b99-82b5-0aefa63fa154 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:32.363564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:32.396384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:32.396999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:32.397303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:32.397566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] PATH_INFO: `/v3/users/83a6816468fd49d8a75f0f090f56da95` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:32.398678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] RBAC: Authorizing `identity:get_user(user_id=83a6816468fd49d8a75f0f090f56da95)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:32.400087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:32.401698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 569/1158] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:32 2022] GET /identity/v3/users/83a6816468fd49d8a75f0f090f56da95 => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:32.407950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:32.514536 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f39d59-475c-4ba0-a14d-926b0738d70a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:32.549623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:32.550225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:32.550465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:32.550818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:32.607026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:32.621720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:32.623751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:32.737086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 590/1159] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:32.743411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:32.843784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c6a60-b225-435a-917a-d6a522172909 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:32.878689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acc597b-5f2d-42c6-a323-e2d038d79f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:32.879356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acc597b-5f2d-42c6-a323-e2d038d79f0b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:32.879630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acc597b-5f2d-42c6-a323-e2d038d79f0b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:32.879991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acc597b-5f2d-42c6-a323-e2d038d79f0b None admin] PATH_INFO: `/v3/projects/c88b3d709c9b437d8775547503907a45` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:32.881490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acc597b-5f2d-42c6-a323-e2d038d79f0b None admin] RBAC: Authorizing `identity:get_project(project_id=c88b3d709c9b437d8775547503907a45)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:32.882917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acc597b-5f2d-42c6-a323-e2d038d79f0b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:32.884612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 570/1160] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:32 2022] GET /identity/v3/projects/c88b3d709c9b437d8775547503907a45 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:32.891132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:32.983820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:32.999840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46536d7f-a4a4-419a-9daf-6bbecbbf2499 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:33.043326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f483175b-ef9a-4294-ac6c-528c393b22ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:33.044316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f483175b-ef9a-4294-ac6c-528c393b22ec None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:33.044792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f483175b-ef9a-4294-ac6c-528c393b22ec None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:33.045272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f483175b-ef9a-4294-ac6c-528c393b22ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:33.046555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f483175b-ef9a-4294-ac6c-528c393b22ec None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=c88b3d709c9b437d8775547503907a45, user.id=83a6816468fd49d8a75f0f090f56da95) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:33.047075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f483175b-ef9a-4294-ac6c-528c393b22ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:33.049611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f483175b-ef9a-4294-ac6c-528c393b22ec None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:33.060359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 591/1161] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:32 2022] GET /identity/v3/role_assignments?user.id=83a6816468fd49d8a75f0f090f56da95&scope.project.id=c88b3d709c9b437d8775547503907a45&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:35.019756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d602bf-4548-4b81-a2b5-24af6794520d None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:35.020280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d602bf-4548-4b81-a2b5-24af6794520d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:35.020280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d602bf-4548-4b81-a2b5-24af6794520d None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:35.021864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 571/1162] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:35.027422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df43de8-40d0-4779-80d3-73264207d4a5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:35.027897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df43de8-40d0-4779-80d3-73264207d4a5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:35.028032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df43de8-40d0-4779-80d3-73264207d4a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:35.047077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3df43de8-40d0-4779-80d3-73264207d4a5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:35.048297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3df43de8-40d0-4779-80d3-73264207d4a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:35.123525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 592/1163] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:35.393866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d602bf-4548-4b81-a2b5-24af6794520d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:35.439428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:35.502357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d602bf-4548-4b81-a2b5-24af6794520d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:35.541978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:35.542583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:35.542860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:35.543135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:35.612896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:35.614424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:35.657250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:35.721076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 572/1164] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:35.727294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df43de8-40d0-4779-80d3-73264207d4a5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:35.806742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:35.827523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df43de8-40d0-4779-80d3-73264207d4a5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:35.867391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f078be3c-b069-4c66-8d35-fc4a3f04ee18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:35.868025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078be3c-b069-4c66-8d35-fc4a3f04ee18 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:35.868303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078be3c-b069-4c66-8d35-fc4a3f04ee18 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:35.868574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f078be3c-b069-4c66-8d35-fc4a3f04ee18 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:35.875321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f078be3c-b069-4c66-8d35-fc4a3f04ee18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:35.876756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f078be3c-b069-4c66-8d35-fc4a3f04ee18 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-f078be3c-b069-4c66-8d35-fc4a3f04ee18 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:35.889104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:35.890508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:35.891372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 593/1165] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:10:35 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:35.897372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:35.999025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24be396f-8432-429f-82bd-233f35f9e960 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:36.034815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03229ad-10e4-4e0f-95b1-3f0150500426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:36.035541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03229ad-10e4-4e0f-95b1-3f0150500426 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:36.035802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03229ad-10e4-4e0f-95b1-3f0150500426 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:36.036062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03229ad-10e4-4e0f-95b1-3f0150500426 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:36.036729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03229ad-10e4-4e0f-95b1-3f0150500426 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:36.037032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03229ad-10e4-4e0f-95b1-3f0150500426 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:36.038421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03229ad-10e4-4e0f-95b1-3f0150500426 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:36.046071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 573/1166] 10.222.0.52 () {56 vars in 1115 bytes} [Sat Aug 27 02:10:35 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:38.037495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d79b14-00d6-422c-a281-504b9ebfab1f None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:38.038068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d79b14-00d6-422c-a281-504b9ebfab1f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:38.038068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d79b14-00d6-422c-a281-504b9ebfab1f None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:38.039268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 594/1167] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:38.045167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fde08-c80e-4987-bfd6-8f297a18c6e1 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:38.045916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fde08-c80e-4987-bfd6-8f297a18c6e1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:38.046072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fde08-c80e-4987-bfd6-8f297a18c6e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:38.055068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:38.067902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-861fde08-c80e-4987-bfd6-8f297a18c6e1 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:38.069115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-861fde08-c80e-4987-bfd6-8f297a18c6e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:38.143246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 574/1168] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:38.409490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d79b14-00d6-422c-a281-504b9ebfab1f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:38.544402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d79b14-00d6-422c-a281-504b9ebfab1f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:38.586270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722fe69-2751-43df-8726-905bc6e201ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:38.587116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722fe69-2751-43df-8726-905bc6e201ec None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:38.587446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722fe69-2751-43df-8726-905bc6e201ec None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:38.587719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722fe69-2751-43df-8726-905bc6e201ec None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:38.588794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f722fe69-2751-43df-8726-905bc6e201ec None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:38.590477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f722fe69-2751-43df-8726-905bc6e201ec None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:38.605835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:38.617794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 595/1169] 10.222.0.52 () {60 vars in 1138 bytes} [Sat Aug 27 02:10:38 2022] POST /identity/v3/domains => generated 267 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:10:40.807359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34d460-6d77-4669-8e5e-eeaa4bfae6e1 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:40.807969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34d460-6d77-4669-8e5e-eeaa4bfae6e1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:40.807969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34d460-6d77-4669-8e5e-eeaa4bfae6e1 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:40.809622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 575/1170] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:40.815636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9500c3a-0915-43fe-a417-4fd7749768a6 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:40.816295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9500c3a-0915-43fe-a417-4fd7749768a6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:40.816295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9500c3a-0915-43fe-a417-4fd7749768a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:40.836971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d9500c3a-0915-43fe-a417-4fd7749768a6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:40.838203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d9500c3a-0915-43fe-a417-4fd7749768a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:40.912447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 596/1171] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:41.181854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34d460-6d77-4669-8e5e-eeaa4bfae6e1 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:41.307004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34d460-6d77-4669-8e5e-eeaa4bfae6e1 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:41.354856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13e103b-882c-47be-bbaa-9951fa6b0cdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:41.355542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e103b-882c-47be-bbaa-9951fa6b0cdb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:41.355846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e103b-882c-47be-bbaa-9951fa6b0cdb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:41.356117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e103b-882c-47be-bbaa-9951fa6b0cdb None admin] PATH_INFO: `/v3/domains/dc6863333ae74d54b6d095ffaddcbc91` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:41.358637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:41.369627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13e103b-882c-47be-bbaa-9951fa6b0cdb None admin] RBAC: Authorizing `identity:get_domain(domain_id=dc6863333ae74d54b6d095ffaddcbc91)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:41.371314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13e103b-882c-47be-bbaa-9951fa6b0cdb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:41.373137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 576/1172] 10.222.0.52 () {56 vars in 1183 bytes} [Sat Aug 27 02:10:41 2022] GET /identity/v3/domains/dc6863333ae74d54b6d095ffaddcbc91 => generated 267 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:41.379987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9500c3a-0915-43fe-a417-4fd7749768a6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:41.527394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9500c3a-0915-43fe-a417-4fd7749768a6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:41.570725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:41.579841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ecd16d-9661-41c1-94ba-3bb790cb80c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:41.580747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ecd16d-9661-41c1-94ba-3bb790cb80c4 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:41.581189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ecd16d-9661-41c1-94ba-3bb790cb80c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:41.581601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ecd16d-9661-41c1-94ba-3bb790cb80c4 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:41.582656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ecd16d-9661-41c1-94ba-3bb790cb80c4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:41.585661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ecd16d-9661-41c1-94ba-3bb790cb80c4 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:41.625681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 597/1173] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:10:41 2022] POST /identity/v3/projects => generated 361 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:10:43.767928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab768c8-4150-48a5-9464-6e1dffd78a0e None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:43.768616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab768c8-4150-48a5-9464-6e1dffd78a0e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:43.768616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab768c8-4150-48a5-9464-6e1dffd78a0e None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:43.769952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 577/1174] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:10:43.776716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b1e5b-433a-4db7-9300-87669a509453 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:43.777217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b1e5b-433a-4db7-9300-87669a509453 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:43.777620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b1e5b-433a-4db7-9300-87669a509453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:43.806668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f60b1e5b-433a-4db7-9300-87669a509453 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:43.807872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f60b1e5b-433a-4db7-9300-87669a509453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:43.880666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 598/1175] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:44.163944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab768c8-4150-48a5-9464-6e1dffd78a0e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:44.275363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab768c8-4150-48a5-9464-6e1dffd78a0e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:44.310692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10508393-6b31-4d11-8d83-e9bd8ca54a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:44.311317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10508393-6b31-4d11-8d83-e9bd8ca54a13 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:44.311594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10508393-6b31-4d11-8d83-e9bd8ca54a13 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:44.311870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10508393-6b31-4d11-8d83-e9bd8ca54a13 None admin] PATH_INFO: `/v3/domains/dc6863333ae74d54b6d095ffaddcbc91` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:44.313039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10508393-6b31-4d11-8d83-e9bd8ca54a13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=dc6863333ae74d54b6d095ffaddcbc91)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:44.314498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10508393-6b31-4d11-8d83-e9bd8ca54a13 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:44.316138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 578/1176] 10.222.0.52 () {56 vars in 1183 bytes} [Sat Aug 27 02:10:44 2022] GET /identity/v3/domains/dc6863333ae74d54b6d095ffaddcbc91 => generated 267 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:44.322819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60b1e5b-433a-4db7-9300-87669a509453 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:44.444744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60b1e5b-433a-4db7-9300-87669a509453 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:44.459876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:44.490423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334716b2-e457-4fb7-a879-e002e751bf26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:44.491419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334716b2-e457-4fb7-a879-e002e751bf26 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:44.491885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334716b2-e457-4fb7-a879-e002e751bf26 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:44.492355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334716b2-e457-4fb7-a879-e002e751bf26 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:44.493547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334716b2-e457-4fb7-a879-e002e751bf26 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:44.496079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334716b2-e457-4fb7-a879-e002e751bf26 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:44.526076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 599/1177] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:10:44 2022] POST /identity/v3/users => generated 311 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:10:46.582506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d9482-66f5-40a1-b7bc-5509bf7cb32f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:46.583331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d9482-66f5-40a1-b7bc-5509bf7cb32f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:46.583331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d9482-66f5-40a1-b7bc-5509bf7cb32f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:46.584584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 579/1178] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:46.590400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583f310-427a-4a3d-a1bd-316eb92c3d10 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:46.590881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583f310-427a-4a3d-a1bd-316eb92c3d10 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:46.591013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583f310-427a-4a3d-a1bd-316eb92c3d10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:46.609463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f583f310-427a-4a3d-a1bd-316eb92c3d10 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:46.610782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f583f310-427a-4a3d-a1bd-316eb92c3d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:46.684760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 600/1179] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:10:46.961207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45d9482-66f5-40a1-b7bc-5509bf7cb32f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:47.030090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:47.072144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45d9482-66f5-40a1-b7bc-5509bf7cb32f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:47.108181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:47.109043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:47.109343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:47.109609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:47.120566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:47.125351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:47.127113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:47.128755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 580/1180] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:10:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:47.133919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583f310-427a-4a3d-a1bd-316eb92c3d10 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:47.256879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:47.263168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583f310-427a-4a3d-a1bd-316eb92c3d10 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:47.322359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:47.323340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:47.323803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:47.324258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:47.325465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:47.325982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:47.328424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:47.342927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 601/1181] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:47.349039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:47.471435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1861e981-dfce-4503-bf93-79ff980ee561 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:47.496054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:47.509899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:47.510441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:47.510706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:47.510970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] PATH_INFO: `/v3/users/4654982693e4427dbe77fbd513ec6301` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:47.525536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] RBAC: Authorizing `identity:get_user(user_id=4654982693e4427dbe77fbd513ec6301)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:47.526982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:47.528657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 581/1182] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:47 2022] GET /identity/v3/users/4654982693e4427dbe77fbd513ec6301 => generated 311 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:47.535003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:47.641634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0f2ecf-8ed4-425a-a30f-4c89a7a21611 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:47.678460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:47.679193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:47.679561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:47.679922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:47.717536 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:47.763354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:47.764699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:47.872806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 602/1183] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:47 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:47.880002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:47.997182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e26360c-f93c-4171-aa4c-841c5fe07780 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:48.033698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44a0df4-79e8-4d5a-b0ff-f0149e9252a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:48.034405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44a0df4-79e8-4d5a-b0ff-f0149e9252a4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:48.034547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44a0df4-79e8-4d5a-b0ff-f0149e9252a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:48.034775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44a0df4-79e8-4d5a-b0ff-f0149e9252a4 None admin] PATH_INFO: `/v3/projects/610ea5632b2644eb9ed0fa9b130719e1` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:48.038024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44a0df4-79e8-4d5a-b0ff-f0149e9252a4 None admin] RBAC: Authorizing `identity:get_project(project_id=610ea5632b2644eb9ed0fa9b130719e1)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:48.039448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44a0df4-79e8-4d5a-b0ff-f0149e9252a4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:48.041142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 582/1184] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:47 2022] GET /identity/v3/projects/610ea5632b2644eb9ed0fa9b130719e1 => generated 361 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:48.050551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:48.139520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:48.203264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809156ef-d7ed-4d67-b92a-8ff6902e37a2 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:48.254850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe662d4-650a-456b-8e7f-27423f9f0612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:48.255715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe662d4-650a-456b-8e7f-27423f9f0612 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:48.256141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe662d4-650a-456b-8e7f-27423f9f0612 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:48.256558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe662d4-650a-456b-8e7f-27423f9f0612 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:48.257754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe662d4-650a-456b-8e7f-27423f9f0612 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=610ea5632b2644eb9ed0fa9b130719e1, user.id=4654982693e4427dbe77fbd513ec6301) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:48.258218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe662d4-650a-456b-8e7f-27423f9f0612 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:48.260523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe662d4-650a-456b-8e7f-27423f9f0612 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:48.269206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 603/1185] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:48 2022] GET /identity/v3/role_assignments?user.id=4654982693e4427dbe77fbd513ec6301&scope.project.id=610ea5632b2644eb9ed0fa9b130719e1&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:50.530846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6934214f-6405-4334-b440-37075a29ab4d None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:50.531669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6934214f-6405-4334-b440-37075a29ab4d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:50.531669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6934214f-6405-4334-b440-37075a29ab4d None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:50.533624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 583/1186] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:50.541609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb92f8a-5804-4bf0-8c88-1f29bb77a330 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:50.542398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb92f8a-5804-4bf0-8c88-1f29bb77a330 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:50.542398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb92f8a-5804-4bf0-8c88-1f29bb77a330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:50.563926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2fb92f8a-5804-4bf0-8c88-1f29bb77a330 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:50.565163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2fb92f8a-5804-4bf0-8c88-1f29bb77a330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:50.644564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 604/1187] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:50.950401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6934214f-6405-4334-b440-37075a29ab4d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:51.016923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:51.077192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6934214f-6405-4334-b440-37075a29ab4d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:51.117951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:51.118896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:51.119305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:51.119669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:51.135377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:51.143191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:51.144421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:10:51.146162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 584/1188] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:10:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:10:51.154118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb92f8a-5804-4bf0-8c88-1f29bb77a330 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:51.236413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:51.297537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb92f8a-5804-4bf0-8c88-1f29bb77a330 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:51.357857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:51.358596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:51.358871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:51.359137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:51.359832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:51.360246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:51.362500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:51.376510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 605/1189] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:51.384462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:51.508590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4267f8f-0ffc-447c-8175-f4d03453aae9 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:51.524566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:51.557654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:51.558398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:51.558538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:51.558795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] PATH_INFO: `/v3/users/4654982693e4427dbe77fbd513ec6301` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:51.559949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] RBAC: Authorizing `identity:get_user(user_id=4654982693e4427dbe77fbd513ec6301)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:51.561410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:51.563014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 585/1190] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:51 2022] GET /identity/v3/users/4654982693e4427dbe77fbd513ec6301 => generated 311 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:51.569480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:51.702231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed5ba0e-ba6c-4f12-bd46-cd6ee07f90ca None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:51.721027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:51.740922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40127ec3-038e-4bd8-a0f3-2d987384425c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:51.741606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40127ec3-038e-4bd8-a0f3-2d987384425c None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:51.741871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40127ec3-038e-4bd8-a0f3-2d987384425c None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:51.742137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40127ec3-038e-4bd8-a0f3-2d987384425c None admin] PATH_INFO: `/v3/projects/610ea5632b2644eb9ed0fa9b130719e1` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:51.743301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40127ec3-038e-4bd8-a0f3-2d987384425c None admin] RBAC: Authorizing `identity:get_project(project_id=610ea5632b2644eb9ed0fa9b130719e1)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:51.744776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40127ec3-038e-4bd8-a0f3-2d987384425c None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:51.746482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 606/1191] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:51 2022] GET /identity/v3/projects/610ea5632b2644eb9ed0fa9b130719e1 => generated 361 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:51.754538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:51.884028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957ddcd-225e-4b30-97c2-04da123bd9d4 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:51.922579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d89633-4622-4792-89d9-c3198887ddc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:51.923261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d89633-4622-4792-89d9-c3198887ddc9 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:51.923532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d89633-4622-4792-89d9-c3198887ddc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:51.923803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d89633-4622-4792-89d9-c3198887ddc9 None admin] PATH_INFO: `/v3/projects/610ea5632b2644eb9ed0fa9b130719e1/users/4654982693e4427dbe77fbd513ec6301/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:51.926088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d89633-4622-4792-89d9-c3198887ddc9 None admin] RBAC: Authorizing `identity:create_grant(project_id=610ea5632b2644eb9ed0fa9b130719e1, user_id=4654982693e4427dbe77fbd513ec6301, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:51.927673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d89633-4622-4792-89d9-c3198887ddc9 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:51.945408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 586/1192] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:10:51 2022] PUT /identity/v3/projects/610ea5632b2644eb9ed0fa9b130719e1/users/4654982693e4427dbe77fbd513ec6301/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:10:53.980328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d543d-828a-4ec0-9fe0-5b373eb54e44 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:53.980911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d543d-828a-4ec0-9fe0-5b373eb54e44 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:53.980911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d543d-828a-4ec0-9fe0-5b373eb54e44 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:53.982450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 607/1193] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:10:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:10:53.989350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ad012-027a-4f71-9e6d-8080b9415b43 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:53.989962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ad012-027a-4f71-9e6d-8080b9415b43 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:53.989962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ad012-027a-4f71-9e6d-8080b9415b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:54.009413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-fd0ad012-027a-4f71-9e6d-8080b9415b43 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:10:54.010679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fd0ad012-027a-4f71-9e6d-8080b9415b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:10:54.063024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:54.141185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 587/1194] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:10:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:10:54.453655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d543d-828a-4ec0-9fe0-5b373eb54e44 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:54.608723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d543d-828a-4ec0-9fe0-5b373eb54e44 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:54.668003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:54.668929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:54.669403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:54.669809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:54.685808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:10:54.693706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:10:54.694812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:10:54.696210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 608/1195] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:10:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:10:54.704509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0ad012-027a-4f71-9e6d-8080b9415b43 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:54.841156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0ad012-027a-4f71-9e6d-8080b9415b43 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:54.886315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:54.887175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:54.887496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:54.887774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:54.888657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:54.888999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:54.890439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:54.900543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 588/1196] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:10:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:54.907094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:54.976062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:55.059337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d48619c-e4c4-45c6-82c4-54f52d792d58 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:55.115871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:55.116860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:55.117265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:55.117675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] PATH_INFO: `/v3/users/4654982693e4427dbe77fbd513ec6301` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:55.119446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] RBAC: Authorizing `identity:get_user(user_id=4654982693e4427dbe77fbd513ec6301)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:55.121739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:55.128703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 609/1197] 10.222.0.52 () {56 vars in 1177 bytes} [Sat Aug 27 02:10:54 2022] GET /identity/v3/users/4654982693e4427dbe77fbd513ec6301 => generated 311 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:10:55.133758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:55.179868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:55.298620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb7b40-fb13-49ff-82e5-aede9e7868a5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:55.345965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:55.346864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:55.347076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:55.347229 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:55.430567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:55.432042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:55.468048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:55.548902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 589/1198] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:10:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:10:55.557242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:55.669697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276208e-3097-4b36-990b-3b144eee919f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:55.683820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:55.713192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e850a11-a729-4e98-9b65-c6f86dc9b2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:55.713956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e850a11-a729-4e98-9b65-c6f86dc9b2ae None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:55.714274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e850a11-a729-4e98-9b65-c6f86dc9b2ae None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:55.714618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e850a11-a729-4e98-9b65-c6f86dc9b2ae None admin] PATH_INFO: `/v3/projects/610ea5632b2644eb9ed0fa9b130719e1` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:55.715894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e850a11-a729-4e98-9b65-c6f86dc9b2ae None admin] RBAC: Authorizing `identity:get_project(project_id=610ea5632b2644eb9ed0fa9b130719e1)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:55.717584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e850a11-a729-4e98-9b65-c6f86dc9b2ae None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:55.719303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 610/1199] 10.222.0.52 () {56 vars in 1186 bytes} [Sat Aug 27 02:10:55 2022] GET /identity/v3/projects/610ea5632b2644eb9ed0fa9b130719e1 => generated 361 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:10:55.725851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:10:55.847883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6329718-2bee-4926-896e-8c3fdb178309 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:10:55.876775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:10:55.887317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a903fe-b4d7-4e25-b313-f16f4af5f165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:10:55.887921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a903fe-b4d7-4e25-b313-f16f4af5f165 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:10:55.888198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a903fe-b4d7-4e25-b313-f16f4af5f165 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:10:55.888465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a903fe-b4d7-4e25-b313-f16f4af5f165 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:10:55.889237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a903fe-b4d7-4e25-b313-f16f4af5f165 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=610ea5632b2644eb9ed0fa9b130719e1, user.id=4654982693e4427dbe77fbd513ec6301) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:10:55.889587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a903fe-b4d7-4e25-b313-f16f4af5f165 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:10:55.891020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a903fe-b4d7-4e25-b313-f16f4af5f165 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:10:55.897496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 590/1200] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:10:55 2022] GET /identity/v3/role_assignments?user.id=4654982693e4427dbe77fbd513ec6301&scope.project.id=610ea5632b2644eb9ed0fa9b130719e1&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:11:05.707804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=10 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:11:05.894682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:11:56.784132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb30855-427b-4601-b37d-1a0ea2988453 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:11:56.784132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb30855-427b-4601-b37d-1a0ea2988453 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:11:56.784132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb30855-427b-4601-b37d-1a0ea2988453 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:11:56.785391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 611/1201] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:11:56 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:11:56.790904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c905aa2-e32b-418a-bf32-89200f3963c3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:11:56.791685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c905aa2-e32b-418a-bf32-89200f3963c3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:11:56.791685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c905aa2-e32b-418a-bf32-89200f3963c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:11:56.818381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2c905aa2-e32b-418a-bf32-89200f3963c3 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:11:56.819991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2c905aa2-e32b-418a-bf32-89200f3963c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:11:56.925187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 591/1202] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:11:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:11:57.249580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb30855-427b-4601-b37d-1a0ea2988453 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:11:57.432137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb30855-427b-4601-b37d-1a0ea2988453 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:11:57.478008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:11:57.478806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:11:57.479288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:11:57.479480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:11:57.498666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:11:57.500727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:11:57.523000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:11:57.524141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:11:57.525360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:11:57.525360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:11:57.525360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:11:57.525360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 612/1203] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:11:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:11:57.533351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c905aa2-e32b-418a-bf32-89200f3963c3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:11:57.675053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c905aa2-e32b-418a-bf32-89200f3963c3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:11:57.713455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:11:57.723803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369948ad-7db6-4357-b006-b5c53abb2da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:11:57.724836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369948ad-7db6-4357-b006-b5c53abb2da1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:11:57.725221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369948ad-7db6-4357-b006-b5c53abb2da1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:11:57.725622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369948ad-7db6-4357-b006-b5c53abb2da1 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:11:57.726423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369948ad-7db6-4357-b006-b5c53abb2da1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:11:57.726877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369948ad-7db6-4357-b006-b5c53abb2da1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:11:57.728557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369948ad-7db6-4357-b006-b5c53abb2da1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:11:57.747989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 592/1204] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:11:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:11:57.749700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:11:57.828906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:11:57.900870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f0f553-9006-43f0-a75f-5591100a4413 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:11:57.944490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd62ddc3-49e5-4677-b37d-95a95c16ac04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:11:57.945231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd62ddc3-49e5-4677-b37d-95a95c16ac04 None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:11:57.945503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd62ddc3-49e5-4677-b37d-95a95c16ac04 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:11:57.945823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd62ddc3-49e5-4677-b37d-95a95c16ac04 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:11:57.946532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd62ddc3-49e5-4677-b37d-95a95c16ac04 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:11:57.948069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd62ddc3-49e5-4677-b37d-95a95c16ac04 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:11:57.978542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 613/1205] 10.222.0.52 () {60 vars in 1132 bytes} [Sat Aug 27 02:11:57 2022] POST /identity/v3/users => generated 251 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:12:00.132983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35255bb-9ebd-4ef0-bf6a-ae46aecd8825 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:00.133566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35255bb-9ebd-4ef0-bf6a-ae46aecd8825 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:00.133566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35255bb-9ebd-4ef0-bf6a-ae46aecd8825 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:00.134918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 593/1206] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:00.140865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b353f9-db5e-430e-90c7-af828448f584 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:00.141385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b353f9-db5e-430e-90c7-af828448f584 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:00.141497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b353f9-db5e-430e-90c7-af828448f584 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:00.160049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c2b353f9-db5e-430e-90c7-af828448f584 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:00.161272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c2b353f9-db5e-430e-90c7-af828448f584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:00.261517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 614/1207] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:00.736800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35255bb-9ebd-4ef0-bf6a-ae46aecd8825 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:00.873694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35255bb-9ebd-4ef0-bf6a-ae46aecd8825 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:00.921328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:00.922178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:00.922433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:00.922742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:00.939007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:12:00.944743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:00.945921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:00.950282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 594/1208] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:12:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:00.952889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b353f9-db5e-430e-90c7-af828448f584 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:01.017450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:01.090692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b353f9-db5e-430e-90c7-af828448f584 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:01.136951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:01.137627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:01.137914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:01.138198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:01.138894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:01.139298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:01.140918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:01.152239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 615/1209] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:12:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:01.162957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:01.227352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:01.286710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7711e6-757b-408f-92ca-69f71cd8acbd None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:01.327961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:01.328584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:01.328866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:01.329265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:01.342728 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:01.342956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:01.361601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:01.363388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:01.364756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:01.364756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:01.364756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:01.364756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 595/1210] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:01.369664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:01.520466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:01.523904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a07b9e4-9df8-4a8a-a78f-d8377bb1e96b None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:01.573491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:01.574122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:01.574429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:01.574714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:01.575457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:01.575788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:01.577294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:01.599789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 616/1211] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:01.603419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:01.777779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ef09f9-8045-4875-8081-b417731ee07f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:01.780598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:01.840720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:01.843008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:01.843578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:01.843946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:01.855472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:01.857253 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:12:01.871483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:01.873564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:01.875127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 596/1212] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:12:01.878570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:02.036648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef437b3-9678-4c89-b788-0fd8bd0023dd None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:02.088711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:02.089438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:02.089675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:02.089924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:02.090871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:02.091030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:02.092464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:02.113546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 617/1213] 10.222.0.52 () {56 vars in 1143 bytes} [Sat Aug 27 02:12:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:12:02.119881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:02.245450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc813c4-0cbe-443b-9a99-ecd1fe82de67 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:02.290498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:02.291136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:02.291391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:02.293176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:02.335305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:02.384847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:02.386608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:02.528288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 597/1214] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:12:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 27 02:12:02.535233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:02.560031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:02.662769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff8e220-8fbd-4e8b-8855-8580004ab7de None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:02.703255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:02.703908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:02.704162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:02.704457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:02.718389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:02.719956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:02.736256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:02.737343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:02.738251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:02.738251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:02.738251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:02.738251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 618/1215] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:12:02.746810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:02.856723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:02.909199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb5923d-aca3-4821-a33b-93766e7ca06b None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:02.963695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:02.964642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:02.964927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:02.965217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:02.965894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:02.966196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:02.967760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:02.987733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 598/1216] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:02.992474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:03.084107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:03.129886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940e810d-ae93-4967-a558-f2415e269454 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:03.182020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:03.182803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:03.183222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:03.183622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:03.192162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:03.193638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:12:03.209141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:03.210523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:03.211614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:03.216519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:03.216906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 619/1217] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:12:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:03.341474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0914d7fa-e424-4990-ae72-23d8c662d52f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:03.371071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:03.386655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f49ea2-e7a5-4544-aeb1-ffc2321e1d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:03.387678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f49ea2-e7a5-4544-aeb1-ffc2321e1d37 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:03.388064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f49ea2-e7a5-4544-aeb1-ffc2321e1d37 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:03.388468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f49ea2-e7a5-4544-aeb1-ffc2321e1d37 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:03.389390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f49ea2-e7a5-4544-aeb1-ffc2321e1d37 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:03.389897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f49ea2-e7a5-4544-aeb1-ffc2321e1d37 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:03.391826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f49ea2-e7a5-4544-aeb1-ffc2321e1d37 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:03.407783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 599/1218] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:12:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:03.414895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:03.550726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b927717d-f185-4235-8785-94ff82b3c55f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:03.584087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:03.591040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af0bd50-a37a-47af-9086-5fa0751a5b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:03.591658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0bd50-a37a-47af-9086-5fa0751a5b78 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:03.591936 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0bd50-a37a-47af-9086-5fa0751a5b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:03.593552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0bd50-a37a-47af-9086-5fa0751a5b78 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:03.593552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af0bd50-a37a-47af-9086-5fa0751a5b78 None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=fa1922df0af7489db403cb6e42a7311b) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:03.593552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af0bd50-a37a-47af-9086-5fa0751a5b78 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:03.594774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af0bd50-a37a-47af-9086-5fa0751a5b78 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:03.601922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 620/1219] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:12:03 2022] GET /identity/v3/role_assignments?user.id=fa1922df0af7489db403cb6e42a7311b&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:05.885012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293e8e7f-d028-4d47-8565-6073ffa7f50c None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:05.885673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293e8e7f-d028-4d47-8565-6073ffa7f50c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:05.885673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293e8e7f-d028-4d47-8565-6073ffa7f50c None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:05.887165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 600/1220] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:05.893178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e406517f-398a-4161-8098-498eb8df6a95 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:05.893711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e406517f-398a-4161-8098-498eb8df6a95 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:05.893792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e406517f-398a-4161-8098-498eb8df6a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:05.916949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e406517f-398a-4161-8098-498eb8df6a95 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:05.918796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e406517f-398a-4161-8098-498eb8df6a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:06.014405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 621/1221] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:06.311727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293e8e7f-d028-4d47-8565-6073ffa7f50c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:06.451016 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293e8e7f-d028-4d47-8565-6073ffa7f50c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:06.490547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:06.491206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:06.491571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:06.491914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:06.509899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:12:06.516370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:06.517570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:06.518574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 601/1222] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:12:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:12:06.524435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e406517f-398a-4161-8098-498eb8df6a95 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:06.636552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e406517f-398a-4161-8098-498eb8df6a95 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:06.666024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:06.680783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:06.681455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:06.681727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:06.681992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:06.682717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] RBAC: Adding query filter params (name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:06.683157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:06.684531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:06.692894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 622/1223] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:12:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:06.700103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:06.751530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:06.811999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5503680-119e-4def-ad55-5eef184fddac None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:06.852015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:06.853083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:06.853543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:06.853845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:06.868943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:06.870620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:06.887565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:06.889806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:06.890694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:06.890694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:06.890694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:06.890694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 602/1224] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:06.894928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:07.014840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb132a-4cbc-4fcd-bade-798af48e8127 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:07.054730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:07.055520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:07.055919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:07.056296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:07.057147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:07.057548 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:07.059229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:07.074579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 623/1225] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:12:07.081600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:07.185999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:07.193780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249fecd1-d8e4-4f0b-a861-a9aa3b722a3e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:07.233299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:07.233991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:07.234328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:07.234600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:07.244202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:07.245960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:12:07.258562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:07.259726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:07.260892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 603/1226] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:07.266041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:07.320232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:07.388574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e7057-a65a-4001-8c12-eaeff8703fb9 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:07.434544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:07.435317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:07.435684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:07.436176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:07.436965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:07.437382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:07.439536 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:07.472494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 624/1227] 10.222.0.52 () {56 vars in 1143 bytes} [Sat Aug 27 02:12:07 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:07.474476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:07.602365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41415eea-b5ae-4daa-bd90-a6ab7defb516 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:07.642201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:07.642882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:07.643157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:07.643423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:07.658567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:07.660270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:07.678297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:07.679482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:07.680553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:07.680553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:07.680553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:07.680553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 604/1228] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:07.687759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:07.798052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91e37e9-36c3-42c6-acee-cb780881b7bb None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:07.801220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:07.836601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:07.837248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:07.837523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:07.837792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:07.838433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:07.838920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:07.840707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:07.853689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 625/1229] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:07.859845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:07.876766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:07.986347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46a5a23-c48b-4a48-94a0-2f2066738666 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:08.026880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:08.027748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:08.028023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:08.028288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:08.036898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:08.038839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:12:08.052077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:08.053356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:08.054288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 605/1230] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:12:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:08.059169 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:08.181248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6791fcca-d4c5-461f-b2fb-246b63359bc6 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:08.220848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3090c43-14a9-47c9-9148-2a2cd3a1c41f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:08.221557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3090c43-14a9-47c9-9148-2a2cd3a1c41f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:08.221905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3090c43-14a9-47c9-9148-2a2cd3a1c41f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:08.222195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3090c43-14a9-47c9-9148-2a2cd3a1c41f None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:08.222919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3090c43-14a9-47c9-9148-2a2cd3a1c41f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:08.223269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3090c43-14a9-47c9-9148-2a2cd3a1c41f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:08.224827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3090c43-14a9-47c9-9148-2a2cd3a1c41f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:08.238719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 626/1231] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:12:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:12:08.245331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:08.333795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:08.373052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80453bd2-8fde-4672-9304-0896b93c98ca None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:08.413701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4e6296-791b-42ad-9216-b22d0ed1a403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:08.414362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e6296-791b-42ad-9216-b22d0ed1a403 None admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:08.414633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e6296-791b-42ad-9216-b22d0ed1a403 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:08.414899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e6296-791b-42ad-9216-b22d0ed1a403 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/fa1922df0af7489db403cb6e42a7311b/roles/439d280788384429bc6d5ff11a303b76` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:08.430695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4e6296-791b-42ad-9216-b22d0ed1a403 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=fa1922df0af7489db403cb6e42a7311b, role_id=439d280788384429bc6d5ff11a303b76)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:08.432912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4e6296-791b-42ad-9216-b22d0ed1a403 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:08.447432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 606/1232] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:12:08 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/fa1922df0af7489db403cb6e42a7311b/roles/439d280788384429bc6d5ff11a303b76 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:12:10.343625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e467fb2-7ea6-4ee7-9064-8c39f39e0bb7 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:10.344224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e467fb2-7ea6-4ee7-9064-8c39f39e0bb7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:10.344224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e467fb2-7ea6-4ee7-9064-8c39f39e0bb7 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:10.345532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 627/1233] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:10.351148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07166988-c55c-43e1-9e7d-c67c2d30add4 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:10.351603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07166988-c55c-43e1-9e7d-c67c2d30add4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:10.351692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07166988-c55c-43e1-9e7d-c67c2d30add4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:10.371716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-07166988-c55c-43e1-9e7d-c67c2d30add4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:10.372971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-07166988-c55c-43e1-9e7d-c67c2d30add4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:10.447452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 607/1234] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:12:10.768095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e467fb2-7ea6-4ee7-9064-8c39f39e0bb7 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:10.796516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:10.892779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e467fb2-7ea6-4ee7-9064-8c39f39e0bb7 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:10.934468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:10.935100 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:10.935368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:10.935626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:10.947532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:12:10.956856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:10.958126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:10.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 628/1235] 10.222.0.52 () {56 vars in 1102 bytes} [Sat Aug 27 02:12:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:10.965351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07166988-c55c-43e1-9e7d-c67c2d30add4 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:11.025522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:11.095248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07166988-c55c-43e1-9e7d-c67c2d30add4 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:11.132871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:11.133509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:11.133784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:11.134055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:11.134729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] RBAC: Adding query filter params (name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:11.135027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:11.136713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:11.145076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 608/1236] 10.222.0.52 () {56 vars in 1103 bytes} [Sat Aug 27 02:12:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:11.151030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:11.246831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:11.267309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd26a375-0129-4d67-8e33-4c17486c1a65 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:11.308106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:11.308832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:11.308941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:11.309146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:11.321214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:11.322758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:11.337883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:11.339326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:11.340249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:11.340249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:11.340249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:11.340249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 629/1237] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:11.345519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:11.466512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b30510-ee98-43d9-9664-75d7c1025343 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:11.476482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:11.502505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:11.503043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:11.503338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:11.503757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:11.504441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:11.504735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:11.506131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:11.518920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 609/1238] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:11.524677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:11.645729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d96213-91b6-4cdc-a477-88e35b518eb0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:11.684529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:11.685171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:11.685440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:11.685712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:11.688565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:11.696180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:11.697741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:12:11.709842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:11.711044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:11.712232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 630/1239] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:11 2022] GET /identity/v3/users/placement => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:11.718883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:11.838197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983490cc-228e-4233-8b53-b77dcaba4321 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:11.884752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:11.885551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:11.886181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:11.886481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:11.887724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:11.888039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:11.889486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:11.908160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 610/1240] 10.222.0.52 () {56 vars in 1143 bytes} [Sat Aug 27 02:12:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:11.914219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:12.037677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2547d0ec-0d52-4093-9041-7d114d2db108 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:12.080418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:12.081198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:12.081650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:12.081942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:12.157110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:12.164857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:12.166738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:12.277551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 631/1241] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:12:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:12:12.284743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:12.318944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:12.403643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c6ab82-b238-4c66-b056-5f77358b3302 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:12.447894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:12.448459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:12.448602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:12.448869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:12.460299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:12.461735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:12.478398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:12.480441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:12.481573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:12.481573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:12.481573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:12.481573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 611/1242] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:12.486561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:12.588996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:12.595437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570fdd05-eed7-4358-85c0-b368c0e4b348 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:12.631983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:12.632590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:12.632864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:12.633147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:12.633843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:12.634125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:12.635543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:12.647383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 632/1243] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:12.654413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:12.748187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:12.772955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee7384-ed3a-4347-9051-47dbab200289 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:12.813489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:12.814618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:12.814900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:12.815162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:12.826226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:12.827980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:12:12.842174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:12.843321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:12.844282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 612/1244] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:12:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:12.849676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:12.959968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1137ffae-8af4-4658-b06e-20aaf2d19cab None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:13.006028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3719f4-a2a5-4534-993c-cb69070cad7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:13.006564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3719f4-a2a5-4534-993c-cb69070cad7b None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:13.006843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3719f4-a2a5-4534-993c-cb69070cad7b None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:13.007139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3719f4-a2a5-4534-993c-cb69070cad7b None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:13.007951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3719f4-a2a5-4534-993c-cb69070cad7b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:13.008446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3719f4-a2a5-4534-993c-cb69070cad7b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:13.010342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3719f4-a2a5-4534-993c-cb69070cad7b None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:13.027008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 633/1245] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:12:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:12:13.033491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:13.153911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283696f6-bf97-4602-b880-4656a5e07ea6 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:13.204055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94ba666-6d3b-445d-b1b6-cbd11a2b5fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:13.204889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ba666-6d3b-445d-b1b6-cbd11a2b5fc1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:13.205386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ba666-6d3b-445d-b1b6-cbd11a2b5fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:13.205873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ba666-6d3b-445d-b1b6-cbd11a2b5fc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:13.206850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94ba666-6d3b-445d-b1b6-cbd11a2b5fc1 None admin] RBAC: Adding query filter params (role.id=439d280788384429bc6d5ff11a303b76, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=fa1922df0af7489db403cb6e42a7311b) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:13.207281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94ba666-6d3b-445d-b1b6-cbd11a2b5fc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:13.209235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94ba666-6d3b-445d-b1b6-cbd11a2b5fc1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:13.217230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 613/1246] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:12:13 2022] GET /identity/v3/role_assignments?user.id=fa1922df0af7489db403cb6e42a7311b&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=439d280788384429bc6d5ff11a303b76 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:15.507379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b7aa3-3fa1-4e69-9ce9-4c5260e5be0b None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:15.508033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b7aa3-3fa1-4e69-9ce9-4c5260e5be0b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:15.508033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b7aa3-3fa1-4e69-9ce9-4c5260e5be0b None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:15.509329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 634/1247] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:15.515590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec685cc8-6d5f-41f0-a2f6-51851b8aa6cf None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:15.516072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec685cc8-6d5f-41f0-a2f6-51851b8aa6cf None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:15.516167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec685cc8-6d5f-41f0-a2f6-51851b8aa6cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:15.523447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:15.536578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ec685cc8-6d5f-41f0-a2f6-51851b8aa6cf None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:15.537838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ec685cc8-6d5f-41f0-a2f6-51851b8aa6cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:15.626713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 614/1248] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:15.896510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60b7aa3-3fa1-4e69-9ce9-4c5260e5be0b None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:15.918522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:16.020872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60b7aa3-3fa1-4e69-9ce9-4c5260e5be0b None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:16.065617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:16.066258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:16.066531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:16.066802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:16.080981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:12:16.088516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:16.090625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:16.092110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 635/1249] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:12:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:16.097080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec685cc8-6d5f-41f0-a2f6-51851b8aa6cf None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:16.223900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec685cc8-6d5f-41f0-a2f6-51851b8aa6cf None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:16.262381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:16.270125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:16.270981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:16.271410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:16.271679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:16.272364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:16.272661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:16.274087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:16.283057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 615/1250] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:12:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:16.289042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:16.388533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:16.424112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dd8410-a322-410d-9598-6a847806055a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:16.466693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:16.467561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:16.468057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:16.468366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:16.484582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:16.486580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:16.509109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:16.510989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:16.512126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:16.512126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:16.512126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:16.512126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 636/1251] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:16.516844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:16.643774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc88ccc-1990-4a08-b765-9259997fc578 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:16.686950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:16.687555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:16.687828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:16.688096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:16.689123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:16.689474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:16.690843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:16.707263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 616/1252] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:16.714827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:16.836953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdedabd8-b327-47d0-8ae1-8f063a6a2436 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:16.868176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:16.876449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:16.877064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:16.877357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:16.877624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:16.888261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:16.889789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:12:16.902048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:16.905800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:16.906951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 637/1253] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:16 2022] GET /identity/v3/users/placement => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:16.910132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:16.974740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:17.027609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d30b08-1ceb-4bf9-af84-86e58bcde692 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:17.073533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:17.074270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:17.074664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:17.074931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:17.075763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:17.076113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:17.078110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:17.101568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 617/1254] 10.222.0.52 () {56 vars in 1143 bytes} [Sat Aug 27 02:12:16 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:17.108867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:17.249424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1704df-59aa-4ac1-b1b5-c77aada23368 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:17.289664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:17.290349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:17.290588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:17.290937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:17.356828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:17.372555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:17.374288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:17.493502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 638/1255] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:12:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:12:17.499573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:17.610497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c423f-17c1-4377-8519-818fd16e44d8 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:17.625521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:17.650567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:17.651290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:17.651568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:17.651837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:17.667356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:17.668406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:17.686489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:17.687567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:17.688536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:17.688536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:17.688536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:17.688536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 618/1256] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:17.694541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:17.802075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:17.816618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81cb26-c458-42c7-a368-6579bf0a3ee2 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:17.858432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:17.859215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:17.859504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:17.859735 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:17.860388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:17.860767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:17.862328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:17.876497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 639/1257] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:17.882557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:18.005229 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57f20f2-db74-447e-aadd-89dbce27f349 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:18.058294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:18.059241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:18.059622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:18.060017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:18.068509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:18.070076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:12:18.082005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:18.083122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:18.084481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 619/1258] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:12:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:18.089290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:18.199945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255b7f44-07c9-4f7f-ba93-37c7f063fcd1 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:18.241620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc4db34-c680-465a-ad64-c834614304f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:18.242395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc4db34-c680-465a-ad64-c834614304f0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:18.242677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc4db34-c680-465a-ad64-c834614304f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:18.242970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc4db34-c680-465a-ad64-c834614304f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:18.243954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc4db34-c680-465a-ad64-c834614304f0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:18.244355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc4db34-c680-465a-ad64-c834614304f0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:18.247148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc4db34-c680-465a-ad64-c834614304f0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:18.265813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 640/1259] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:12:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:18.272774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:18.300113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:18.389473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488d4e4d-b460-4e91-a67e-10bedd83489f None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:18.452342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b0e4cb-e65b-48fa-b1da-c87ee9994f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:18.453081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b0e4cb-e65b-48fa-b1da-c87ee9994f98 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:18.453445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b0e4cb-e65b-48fa-b1da-c87ee9994f98 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:18.453821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b0e4cb-e65b-48fa-b1da-c87ee9994f98 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:18.454616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b0e4cb-e65b-48fa-b1da-c87ee9994f98 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=fa1922df0af7489db403cb6e42a7311b) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:18.454950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b0e4cb-e65b-48fa-b1da-c87ee9994f98 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:18.456786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b0e4cb-e65b-48fa-b1da-c87ee9994f98 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:18.466140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 620/1260] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:12:18 2022] GET /identity/v3/role_assignments?user.id=fa1922df0af7489db403cb6e42a7311b&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=b83e461b8f6643fba45f427d28498c41 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:12:20.420626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8a7cb-9fe7-431d-be0c-3ed0a23d8be6 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:20.421237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8a7cb-9fe7-431d-be0c-3ed0a23d8be6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:20.421358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f8a7cb-9fe7-431d-be0c-3ed0a23d8be6 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:20.423002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 641/1261] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:20.430077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44b37c-87d0-445c-aa33-d3c4d4d9d44e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:20.430641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44b37c-87d0-445c-aa33-d3c4d4d9d44e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:20.430731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44b37c-87d0-445c-aa33-d3c4d4d9d44e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:20.450539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4a44b37c-87d0-445c-aa33-d3c4d4d9d44e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:20.451748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4a44b37c-87d0-445c-aa33-d3c4d4d9d44e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:20.528137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 621/1262] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 27 02:12:20.797968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8a7cb-9fe7-431d-be0c-3ed0a23d8be6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:20.810505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:20.905411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f8a7cb-9fe7-431d-be0c-3ed0a23d8be6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:20.940564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:20.941186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:20.941485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:20.941749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:20.952783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:12:20.957648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:20.958721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:20.959903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 642/1263] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:12:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:20.965464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44b37c-87d0-445c-aa33-d3c4d4d9d44e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:20.994620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:21.079138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44b37c-87d0-445c-aa33-d3c4d4d9d44e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:21.118635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:21.119377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:21.119636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:21.119888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:21.120572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:21.120871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:21.122254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:21.130512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 622/1264] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:12:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:21.136516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:21.215669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:21.256224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7995b84b-36d2-44f6-a0fb-e18ed131cd19 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:21.293190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:21.293825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:21.294099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:21.294368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:21.310520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:21.311944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:21.327378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:21.328460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:21.329571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:21.329571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:21.329571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:21.329571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 643/1265] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:21.334937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:21.440149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:21.461574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b01dc9c-1bd0-4b6d-a413-376299409678 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:21.500342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:21.500877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:21.501160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:21.501439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:21.502102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:21.502428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:21.503795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:21.516747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 623/1266] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:21.523594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:21.635373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f3eb23-3445-4915-8a81-68bd61662239 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:21.656259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:21.673706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:21.674340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:21.674611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:21.674879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:21.684106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:21.685578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:12:21.697160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:21.698413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:21.699934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 644/1267] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:21 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:21.707539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:21.813944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639fd686-8927-47d0-baf1-6dff14e088cb None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:21.857843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:21.858476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:21.858752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:21.859032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:21.859782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:21.860084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:21.861572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:21.871345 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:21.880413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 624/1268] 10.222.0.52 () {56 vars in 1143 bytes} [Sat Aug 27 02:12:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:12:21.887532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:22.004959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6a63c0-37f1-4fdb-97cd-7e6900a7345f None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:22.042012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:22.042635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:22.042902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:22.043162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:22.054692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:22.056152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:22.071875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:22.073421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:22.074409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:22.074409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:22.074409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:22.074409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 645/1269] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:22.081075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:22.189373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f94d59b-1cce-4aa5-baf3-2a13e11c161e None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:22.233534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:22.234846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:22.235295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:22.235710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:22.236698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:22.237001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:22.238938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:22.256054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 625/1270] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:22.264626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:22.305297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:22.391173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5823e61c-b372-4f38-9b6a-15f6163469ac None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:22.434027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:22.434671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:22.434942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:22.435207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:22.443333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:22.444775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:12:22.456719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:22.457837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:22.458960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 646/1271] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:12:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:22.464837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:22.526150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:22.580998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a8dad-ef79-42e1-bdf1-616d6c31fe43 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:22.623851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff63858c-d38b-4bd2-bc55-5f070869829c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:22.624966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff63858c-d38b-4bd2-bc55-5f070869829c None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:22.625267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff63858c-d38b-4bd2-bc55-5f070869829c None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:22.625527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff63858c-d38b-4bd2-bc55-5f070869829c None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:22.626226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff63858c-d38b-4bd2-bc55-5f070869829c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:22.626527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff63858c-d38b-4bd2-bc55-5f070869829c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:22.627959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff63858c-d38b-4bd2-bc55-5f070869829c None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:22.641825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 626/1272] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:12:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:22.651355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:22.761505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:22.771730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c8b305-9502-4863-909b-0316a66d86e0 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:22.807878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c57b57c-beb3-4a4a-8cc6-81da6ce37ef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:22.808505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c57b57c-beb3-4a4a-8cc6-81da6ce37ef6 None admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:22.808796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c57b57c-beb3-4a4a-8cc6-81da6ce37ef6 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:22.809057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c57b57c-beb3-4a4a-8cc6-81da6ce37ef6 None admin] PATH_INFO: `/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/fa1922df0af7489db403cb6e42a7311b/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:22.811020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c57b57c-beb3-4a4a-8cc6-81da6ce37ef6 None admin] RBAC: Authorizing `identity:create_grant(project_id=89716fa846034fe3b8d98abb238e5f22, user_id=fa1922df0af7489db403cb6e42a7311b, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:22.812445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c57b57c-beb3-4a4a-8cc6-81da6ce37ef6 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:22.828615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 647/1273] 10.222.0.52 () {58 vars in 1439 bytes} [Sat Aug 27 02:12:22 2022] PUT /identity/v3/projects/89716fa846034fe3b8d98abb238e5f22/users/fa1922df0af7489db403cb6e42a7311b/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:12:24.772336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ebf14-9921-45ac-80e3-28baaf6a31b9 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:24.772907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ebf14-9921-45ac-80e3-28baaf6a31b9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:24.772907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ebf14-9921-45ac-80e3-28baaf6a31b9 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:24.774804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 627/1274] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:24.781192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa1ab0-b1ab-47a8-a6f3-1dbd3c977ec5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:24.781659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa1ab0-b1ab-47a8-a6f3-1dbd3c977ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:24.781777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffa1ab0-b1ab-47a8-a6f3-1dbd3c977ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:24.801273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-fffa1ab0-b1ab-47a8-a6f3-1dbd3c977ec5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:24.802509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fffa1ab0-b1ab-47a8-a6f3-1dbd3c977ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:24.875426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 648/1275] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:25.141745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ebf14-9921-45ac-80e3-28baaf6a31b9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:25.249824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ebf14-9921-45ac-80e3-28baaf6a31b9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:25.267404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:25.285155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:25.285768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:25.286045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:25.286312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:25.296654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 27 02:12:25.301496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:25.302851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:25.303864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 628/1276] 10.222.0.52 () {56 vars in 1096 bytes} [Sat Aug 27 02:12:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:25.309672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffa1ab0-b1ab-47a8-a6f3-1dbd3c977ec5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:25.405635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:25.435988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffa1ab0-b1ab-47a8-a6f3-1dbd3c977ec5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:25.484277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:25.484893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:25.485193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:25.485464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:25.486139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:25.486437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:25.487971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:25.498743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 649/1277] 10.222.0.52 () {56 vars in 1099 bytes} [Sat Aug 27 02:12:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:12:25.506182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:25.615135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcba2f8-c8a7-4b5d-be7c-43369052557a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:25.649440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:25.650095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:25.650391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:25.650658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:25.661239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:25.662643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:25.678449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:25.679731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:25.680666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:25.680666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:25.680666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:25.680666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 629/1278] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:25.686321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:25.797701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8a82c7-3e1e-4cd0-bfb9-12bc11bc404d None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:25.833401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:25.834002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:25.834275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:25.834541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:25.835180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:25.835478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:25.836846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:25.848980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 650/1279] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:25.856762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:25.896985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:25.975729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fea566-4fe6-4b79-aefc-10daf6211eae None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:26.014323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:26.014949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:26.015219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:26.015509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:26.023289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:26.024802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 27 02:12:26.037616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:26.039460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:26.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 630/1280] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:25 2022] GET /identity/v3/users/placement => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:26.045162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:26.048964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:26.159818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc225de9-5d4a-4b0d-84ef-eed69b530649 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:26.207241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:26.207846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:26.208088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:26.208373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:26.209111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:26.209414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:26.210861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:26.231959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 651/1281] 10.222.0.52 () {56 vars in 1143 bytes} [Sat Aug 27 02:12:26 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:26.237939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:26.346054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:26.356184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a605ae-94b2-4c14-bdfb-a539d1946ad5 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:26.397888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:26.398513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:26.398783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:26.399046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:26.484593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:26.486010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:26.597415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:26.611459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 631/1282] 10.222.0.52 () {58 vars in 1282 bytes} [Sat Aug 27 02:12:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 27 02:12:26.619976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:26.692818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:26.741628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f764b-22cf-4dcc-b81a-6602ba905e65 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:26.779105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:26.779740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:26.780011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:26.780287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:26.791693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:26.793996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 27 02:12:26.814260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:26.815869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 27 02:12:26.816962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 27 02:12:26.816962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 27 02:12:26.816962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:26.816962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 652/1283] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:12:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:26.824664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:26.939116 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda1c8ec-295d-442a-a200-64070ee732e1 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:26.974833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:26.975433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:26.975704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:26.975970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:26.976621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:26.976924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:26.978400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:26.991862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 632/1284] 10.222.0.52 () {56 vars in 1109 bytes} [Sat Aug 27 02:12:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:26.998205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:27.114463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22a0a9-ff0d-4246-863a-18bebf156b59 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:27.135040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:27.163932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:27.164584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:27.164895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:27.165241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:27.173076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:27.174918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 27 02:12:27.191499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:27.192608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: ERROR keystone.server.flask.application Aug 27 02:12:27.194043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 653/1285] 10.222.0.52 () {56 vars in 1111 bytes} [Sat Aug 27 02:12:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:27.198866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:27.257607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:27.318148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb2769-6ea7-4a65-87d0-edc9f44e198a None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:27.353638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7775cb56-8f14-4055-a383-384456d0cdd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:27.354411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7775cb56-8f14-4055-a383-384456d0cdd8 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:27.354732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7775cb56-8f14-4055-a383-384456d0cdd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:27.354998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7775cb56-8f14-4055-a383-384456d0cdd8 None admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:27.355704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7775cb56-8f14-4055-a383-384456d0cdd8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:27.356012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7775cb56-8f14-4055-a383-384456d0cdd8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:27.357465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7775cb56-8f14-4055-a383-384456d0cdd8 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:27.370063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 633/1286] 10.222.0.52 () {56 vars in 1148 bytes} [Sat Aug 27 02:12:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:27.377981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:27.509367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b399f0f0-c745-401e-8e76-24a579c9a37a None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:27.553010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01017ac-b97b-4399-aad6-b22c57a81b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:27.553687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01017ac-b97b-4399-aad6-b22c57a81b53 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:27.553962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01017ac-b97b-4399-aad6-b22c57a81b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:27.554233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01017ac-b97b-4399-aad6-b22c57a81b53 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:27.554974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01017ac-b97b-4399-aad6-b22c57a81b53 None admin] RBAC: Adding query filter params (role.id=b83e461b8f6643fba45f427d28498c41, scope.project.id=89716fa846034fe3b8d98abb238e5f22, user.id=fa1922df0af7489db403cb6e42a7311b) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:27.555240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01017ac-b97b-4399-aad6-b22c57a81b53 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:27.556808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01017ac-b97b-4399-aad6-b22c57a81b53 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:27.565822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 654/1287] 10.222.0.52 () {56 vars in 1374 bytes} [Sat Aug 27 02:12:27 2022] GET /identity/v3/role_assignments?user.id=fa1922df0af7489db403cb6e42a7311b&scope.project.id=89716fa846034fe3b8d98abb238e5f22&role.id=b83e461b8f6643fba45f427d28498c41 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:29.522150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246d2a9-3dc4-41ad-bd46-1b5760ea6403 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:29.522965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246d2a9-3dc4-41ad-bd46-1b5760ea6403 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:29.523117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246d2a9-3dc4-41ad-bd46-1b5760ea6403 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:29.524510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 634/1288] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:29.530445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f659c541-0ee2-4099-923f-a30346b2297c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:29.531251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f659c541-0ee2-4099-923f-a30346b2297c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:29.531415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f659c541-0ee2-4099-923f-a30346b2297c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:29.552642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f659c541-0ee2-4099-923f-a30346b2297c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:29.554143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f659c541-0ee2-4099-923f-a30346b2297c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:29.592495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:29.646597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 655/1289] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:29.939062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d246d2a9-3dc4-41ad-bd46-1b5760ea6403 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:30.055759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d246d2a9-3dc4-41ad-bd46-1b5760ea6403 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:30.067592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:30.094542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:30.095171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:30.095650 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:30.095789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:30.096297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:30.097693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:30.106878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:30.108112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:30.109056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 635/1290] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:12:29 2022] GET /identity/v3/services/placement => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:30.114811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f659c541-0ee2-4099-923f-a30346b2297c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:30.224735 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f659c541-0ee2-4099-923f-a30346b2297c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:30.271825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88fee32-5644-4c85-a775-fda6912c10f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:30.272444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88fee32-5644-4c85-a775-fda6912c10f5 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:30.272713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88fee32-5644-4c85-a775-fda6912c10f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:30.272978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88fee32-5644-4c85-a775-fda6912c10f5 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:30.273652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88fee32-5644-4c85-a775-fda6912c10f5 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:30.273952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88fee32-5644-4c85-a775-fda6912c10f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:30.275629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88fee32-5644-4c85-a775-fda6912c10f5 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:30.286125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 656/1291] 10.222.0.52 () {56 vars in 1116 bytes} [Sat Aug 27 02:12:30 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:30.295080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:30.441926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d23772-8fee-49d5-b4f7-9a3352c39047 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:30.496844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbcfdd7-337a-4040-acc9-9c06913dc508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:30.497408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbcfdd7-337a-4040-acc9-9c06913dc508 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:30.497660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbcfdd7-337a-4040-acc9-9c06913dc508 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:30.497947 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbcfdd7-337a-4040-acc9-9c06913dc508 None admin] PATH_INFO: `/v3/services` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:30.498599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbcfdd7-337a-4040-acc9-9c06913dc508 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:30.498890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbcfdd7-337a-4040-acc9-9c06913dc508 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:30.500268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbcfdd7-337a-4040-acc9-9c06913dc508 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:30.506494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 636/1292] 10.222.0.52 () {56 vars in 1116 bytes} [Sat Aug 27 02:12:30 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:32.422572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff277d-c2b5-4cc1-953a-90f61e9b7af5 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:32.423157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff277d-c2b5-4cc1-953a-90f61e9b7af5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:32.423157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff277d-c2b5-4cc1-953a-90f61e9b7af5 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:32.424425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 657/1293] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:32.429887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d775b763-d763-4932-a377-0904253694d1 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:32.430316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d775b763-d763-4932-a377-0904253694d1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:32.430597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d775b763-d763-4932-a377-0904253694d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:32.451737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-d775b763-d763-4932-a377-0904253694d1 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:32.452949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d775b763-d763-4932-a377-0904253694d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:32.485008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:32.526922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 637/1294] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:32.796668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ff277d-c2b5-4cc1-953a-90f61e9b7af5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:32.829083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:32.902504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ff277d-c2b5-4cc1-953a-90f61e9b7af5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:32.938707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb375be6-8170-42f9-acda-007a4c1f8ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:32.939308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb375be6-8170-42f9-acda-007a4c1f8ade None admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:32.939578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb375be6-8170-42f9-acda-007a4c1f8ade None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:32.939841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb375be6-8170-42f9-acda-007a4c1f8ade None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:32.940485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb375be6-8170-42f9-acda-007a4c1f8ade None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:32.941876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb375be6-8170-42f9-acda-007a4c1f8ade None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:32.964688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 658/1295] 10.222.0.52 () {60 vars in 1141 bytes} [Sat Aug 27 02:12:32 2022] POST /identity/v3/services => generated 246 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:12:35.427821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b7a98e-c11d-4475-9071-32a572c97237 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:35.428637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b7a98e-c11d-4475-9071-32a572c97237 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:35.428637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b7a98e-c11d-4475-9071-32a572c97237 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:35.430202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 638/1296] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:35.439109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476d96ce-8259-4a9e-bccf-af4a0c4630df None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:35.440569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476d96ce-8259-4a9e-bccf-af4a0c4630df None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:35.440569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476d96ce-8259-4a9e-bccf-af4a0c4630df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:35.463639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-476d96ce-8259-4a9e-bccf-af4a0c4630df None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:35.465177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-476d96ce-8259-4a9e-bccf-af4a0c4630df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:35.554187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 659/1297] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:35 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:35.833162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b7a98e-c11d-4475-9071-32a572c97237 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:35.944234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b7a98e-c11d-4475-9071-32a572c97237 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:35.988900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:35.989787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:35.990220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:35.990681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:35.991635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:35.994187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:36.009180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:36.010568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:36.011777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 639/1298] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:12:35 2022] GET /identity/v3/services/placement => generated 90 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 27 02:12:36.020137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476d96ce-8259-4a9e-bccf-af4a0c4630df None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:36.049255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:36.176468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476d96ce-8259-4a9e-bccf-af4a0c4630df None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:36.235068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:36.235959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:36.236365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:36.236778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:36.237789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:36.238235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:36.240500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:36.250049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 660/1299] 10.222.0.52 () {56 vars in 1116 bytes} [Sat Aug 27 02:12:36 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:36.260336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:36.314350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:36.405436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c717b24-ac0d-4485-9feb-2cc64b8d8d24 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:36.440529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550e60e-8179-4472-a484-a639d0f6f57d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:36.441287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550e60e-8179-4472-a484-a639d0f6f57d None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:36.441416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550e60e-8179-4472-a484-a639d0f6f57d None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:36.441675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550e60e-8179-4472-a484-a639d0f6f57d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:36.442385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0550e60e-8179-4472-a484-a639d0f6f57d None admin] RBAC: Adding query filter params (interface=public, service_id=f05877d3af2b47829ba905c9131499c9, region_id=RegionOne) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:36.442686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0550e60e-8179-4472-a484-a639d0f6f57d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:36.444035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0550e60e-8179-4472-a484-a639d0f6f57d None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:36.449989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 640/1300] 10.222.0.52 () {56 vars in 1251 bytes} [Sat Aug 27 02:12:36 2022] GET /identity/v3/endpoints?service_id=f05877d3af2b47829ba905c9131499c9&interface=public®ion_id=RegionOne => generated 194 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:36.455897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:36.544017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:36.572640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56775d3d-c981-4ffd-9875-754d81d68648 None admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:36.628429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba398595-6f39-4989-9db6-ac1e09746c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:36.629467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba398595-6f39-4989-9db6-ac1e09746c82 None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:36.629938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba398595-6f39-4989-9db6-ac1e09746c82 None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:36.630397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba398595-6f39-4989-9db6-ac1e09746c82 None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:36.631449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba398595-6f39-4989-9db6-ac1e09746c82 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:36.634142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba398595-6f39-4989-9db6-ac1e09746c82 None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:36.646930 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 661/1301] 10.222.0.52 () {56 vars in 1087 bytes} [Sat Aug 27 02:12:36 2022] GET /identity/v3/services => generated 2191 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:12:38.623635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e45951-73e0-4256-b3ea-1d057db01841 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:38.624242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e45951-73e0-4256-b3ea-1d057db01841 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:38.624242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e45951-73e0-4256-b3ea-1d057db01841 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:38.625551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 641/1302] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:12:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:12:38.631244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc266ad6-9cca-46c3-82b2-702a612c87c6 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:38.631692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc266ad6-9cca-46c3-82b2-702a612c87c6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:38.631847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc266ad6-9cca-46c3-82b2-702a612c87c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:38.651152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-dc266ad6-9cca-46c3-82b2-702a612c87c6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:12:38.652340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dc266ad6-9cca-46c3-82b2-702a612c87c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:12:38.754802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 662/1303] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:12:38 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 27 02:12:39.018726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e45951-73e0-4256-b3ea-1d057db01841 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:39.125148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:39.132783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e45951-73e0-4256-b3ea-1d057db01841 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:39.167398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:39.168070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:39.168393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:39.168705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:39.169404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:39.170818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 27 02:12:39.179858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 27 02:12:39.181914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.get_or_create( Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application with Lock( Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self._enter() Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application return self.creator() Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application created_value = creator( Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: ERROR keystone.server.flask.application Aug 27 02:12:39.183487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 642/1304] 10.222.0.52 () {56 vars in 1117 bytes} [Sat Aug 27 02:12:39 2022] GET /identity/v3/services/placement => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 27 02:12:39.189057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc266ad6-9cca-46c3-82b2-702a612c87c6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:39.269031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:39.293576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc266ad6-9cca-46c3-82b2-702a612c87c6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:39.329275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11403e28-4b5e-49f2-9200-85f9692cff8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:39.329951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11403e28-4b5e-49f2-9200-85f9692cff8f None admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:39.330269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11403e28-4b5e-49f2-9200-85f9692cff8f None admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:39.330574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11403e28-4b5e-49f2-9200-85f9692cff8f None admin] PATH_INFO: `/v3/services` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:39.331245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11403e28-4b5e-49f2-9200-85f9692cff8f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:12:39.331579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11403e28-4b5e-49f2-9200-85f9692cff8f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:39.333023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11403e28-4b5e-49f2-9200-85f9692cff8f None admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:39.338701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 663/1305] 10.222.0.52 () {56 vars in 1116 bytes} [Sat Aug 27 02:12:39 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:12:39.345286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:12:39.483180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7d6eef-e846-43b8-8017-9d20583eea49 None admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:12:39.518319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a920713e-e27a-41b1-8544-f0305ccd61ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:12:39.518952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920713e-e27a-41b1-8544-f0305ccd61ca None admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:12:39.519287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920713e-e27a-41b1-8544-f0305ccd61ca None admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:12:39.519478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920713e-e27a-41b1-8544-f0305ccd61ca None admin] PATH_INFO: `/v3/endpoints` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:12:39.520158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a920713e-e27a-41b1-8544-f0305ccd61ca None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:12:39.521731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a920713e-e27a-41b1-8544-f0305ccd61ca None admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:12:39.542950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 643/1306] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:12:39 2022] POST /identity/v3/endpoints => generated 331 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:12:49.335233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=30 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:12:49.537751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:13:55.822912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea95fb7-9f6f-4de5-9523-0c6862931106 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:13:55.824179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea95fb7-9f6f-4de5-9523-0c6862931106 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:13:55.824872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea95fb7-9f6f-4de5-9523-0c6862931106 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:13:55.828094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 664/1307] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:13:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:13:55.836827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e19449-74f5-41c3-afe0-e838b6a85268 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:13:55.838555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e19449-74f5-41c3-afe0-e838b6a85268 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:13:55.839395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e19449-74f5-41c3-afe0-e838b6a85268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:13:56.080868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-25e19449-74f5-41c3-afe0-e838b6a85268 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:13:56.082578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-25e19449-74f5-41c3-afe0-e838b6a85268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:13:56.538317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 644/1308] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:13:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 703 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:13:57.035307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59f3e8-ded5-40b1-b938-ba0d3cb3784d None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:13:57.036491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59f3e8-ded5-40b1-b938-ba0d3cb3784d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:13:57.036722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59f3e8-ded5-40b1-b938-ba0d3cb3784d None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:13:57.040263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 665/1309] 10.222.0.52 () {54 vars in 965 bytes} [Sat Aug 27 02:13:57 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:13:57.052248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35965909-9e05-4f4d-a008-29839748e45e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:13:57.053738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35965909-9e05-4f4d-a008-29839748e45e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:13:57.054471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35965909-9e05-4f4d-a008-29839748e45e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:13:57.219664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-35965909-9e05-4f4d-a008-29839748e45e None None] MFA Rules not processed for user `56aaa029a9c542e0bc4658c5cd55c2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:13:57.221502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-35965909-9e05-4f4d-a008-29839748e45e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:13:57.262801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:13:57.362430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 645/1310] 10.222.0.52 () {58 vars in 1062 bytes} [Sat Aug 27 02:13:57 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:13:57.380474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59f3e8-ded5-40b1-b938-ba0d3cb3784d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:13:57.384832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6a59f3e8-ded5-40b1-b938-ba0d3cb3784d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:13:57.592208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59f3e8-ded5-40b1-b938-ba0d3cb3784d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:13:57.739295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b3f14f-4f15-4244-ae4a-a5e2ea301b13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:13:57.740711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b3f14f-4f15-4244-ae4a-a5e2ea301b13 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:13:57.741417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b3f14f-4f15-4244-ae4a-a5e2ea301b13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:13:57.742069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b3f14f-4f15-4244-ae4a-a5e2ea301b13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:13:57.745847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-41b3f14f-4f15-4244-ae4a-a5e2ea301b13 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:13:57.864505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:13:57.958937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b3f14f-4f15-4244-ae4a-a5e2ea301b13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:13:57.962343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b3f14f-4f15-4244-ae4a-a5e2ea301b13 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:13:58.004492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 666/1311] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:13:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 625 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:00.995033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39644fe0-ee1c-4f83-9bd1-ca41ecbd4d75 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:00.995861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39644fe0-ee1c-4f83-9bd1-ca41ecbd4d75 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:00.995861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39644fe0-ee1c-4f83-9bd1-ca41ecbd4d75 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:00.996811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 646/1312] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:14:01.002590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade6a05b-e1bc-4516-bb03-d51c13918b53 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:01.003193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade6a05b-e1bc-4516-bb03-d51c13918b53 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:01.003193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade6a05b-e1bc-4516-bb03-d51c13918b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:01.093756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ade6a05b-e1bc-4516-bb03-d51c13918b53 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:01.095641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ade6a05b-e1bc-4516-bb03-d51c13918b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:01.220333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 667/1313] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:14:01.510261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39644fe0-ee1c-4f83-9bd1-ca41ecbd4d75 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:01.513007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-39644fe0-ee1c-4f83-9bd1-ca41ecbd4d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:01.663065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39644fe0-ee1c-4f83-9bd1-ca41ecbd4d75 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:01.771944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:01.775467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83dbde7-859e-4392-8541-579ab5decbeb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:01.777680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83dbde7-859e-4392-8541-579ab5decbeb service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:01.777680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83dbde7-859e-4392-8541-579ab5decbeb service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:01.777680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83dbde7-859e-4392-8541-579ab5decbeb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:01.779802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a83dbde7-859e-4392-8541-579ab5decbeb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:01.906131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83dbde7-859e-4392-8541-579ab5decbeb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:01.907590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83dbde7-859e-4392-8541-579ab5decbeb service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:01.928556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 647/1314] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:04.567415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a609b47-3666-422e-a24b-184f7775d11e None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:04.568031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a609b47-3666-422e-a24b-184f7775d11e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:04.568031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a609b47-3666-422e-a24b-184f7775d11e None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:04.569392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 668/1315] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:04.576926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91856e87-3dde-4c85-8be9-8e4b7005c585 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:04.577512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91856e87-3dde-4c85-8be9-8e4b7005c585 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:04.577512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91856e87-3dde-4c85-8be9-8e4b7005c585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:04.691344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-91856e87-3dde-4c85-8be9-8e4b7005c585 None None] MFA Rules not processed for user `1c1b7dba7cb84f5c89be5d6f9d6f60a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:04.692595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-91856e87-3dde-4c85-8be9-8e4b7005c585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:04.768088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:04.825788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 648/1316] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:04 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:05.139252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a609b47-3666-422e-a24b-184f7775d11e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:05.140803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2a609b47-3666-422e-a24b-184f7775d11e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:05.156599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:05.288885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a609b47-3666-422e-a24b-184f7775d11e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:05.392487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81744a7-40f8-4fd8-87a7-d4bec186eb5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:05.393370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81744a7-40f8-4fd8-87a7-d4bec186eb5e service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:05.393968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81744a7-40f8-4fd8-87a7-d4bec186eb5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:05.394203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81744a7-40f8-4fd8-87a7-d4bec186eb5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:05.396718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a81744a7-40f8-4fd8-87a7-d4bec186eb5e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:05.488404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:05.551722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81744a7-40f8-4fd8-87a7-d4bec186eb5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:05.553329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81744a7-40f8-4fd8-87a7-d4bec186eb5e service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:05.578240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 669/1317] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:05 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:14:08.514864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20710f19-47cf-486c-a39a-0ca8b055ce2c None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:08.515538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20710f19-47cf-486c-a39a-0ca8b055ce2c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:08.515538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20710f19-47cf-486c-a39a-0ca8b055ce2c None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:08.516739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 649/1318] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:08.524049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c79e03c-8d03-458e-adb6-43bffdc70d9d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:08.524600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c79e03c-8d03-458e-adb6-43bffdc70d9d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:08.524998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c79e03c-8d03-458e-adb6-43bffdc70d9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:08.619170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6c79e03c-8d03-458e-adb6-43bffdc70d9d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:08.620708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6c79e03c-8d03-458e-adb6-43bffdc70d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:08.765759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 670/1319] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:08 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:09.085521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570d19d-a152-4774-9594-e0c1f692f3f9 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:09.085937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570d19d-a152-4774-9594-e0c1f692f3f9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:09.086439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570d19d-a152-4774-9594-e0c1f692f3f9 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:09.088523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 650/1320] 10.222.0.52 () {54 vars in 965 bytes} [Sat Aug 27 02:14:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:09.095063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043e5d7-b547-4c57-a44b-629e0054993a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:09.095351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043e5d7-b547-4c57-a44b-629e0054993a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:09.095631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043e5d7-b547-4c57-a44b-629e0054993a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:09.120151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:09.188179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-7043e5d7-b547-4c57-a44b-629e0054993a None None] MFA Rules not processed for user `56aaa029a9c542e0bc4658c5cd55c2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:09.190321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7043e5d7-b547-4c57-a44b-629e0054993a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:09.288598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 671/1321] 10.222.0.52 () {58 vars in 1062 bytes} [Sat Aug 27 02:14:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:09.302307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1570d19d-a152-4774-9594-e0c1f692f3f9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:09.303849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1570d19d-a152-4774-9594-e0c1f692f3f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:09.433172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1570d19d-a152-4774-9594-e0c1f692f3f9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:09.525734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3eb34c5-382a-4e76-bba2-e29a3e3e706d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:09.526695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eb34c5-382a-4e76-bba2-e29a3e3e706d service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:09.527180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eb34c5-382a-4e76-bba2-e29a3e3e706d service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:09.527647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eb34c5-382a-4e76-bba2-e29a3e3e706d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:09.530672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b3eb34c5-382a-4e76-bba2-e29a3e3e706d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:09.543539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:09.661054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3eb34c5-382a-4e76-bba2-e29a3e3e706d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:09.662650 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3eb34c5-382a-4e76-bba2-e29a3e3e706d service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:09.682003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 651/1322] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:14:13.690786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadee800-b038-4498-b807-93481eb28ec4 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:13.691378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadee800-b038-4498-b807-93481eb28ec4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:13.691378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadee800-b038-4498-b807-93481eb28ec4 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:13.692685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 672/1323] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:13.699445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746ce08-2ffb-493d-ab1a-fe3c8e45d27f None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:13.699924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746ce08-2ffb-493d-ab1a-fe3c8e45d27f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:13.700039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746ce08-2ffb-493d-ab1a-fe3c8e45d27f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:13.791997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1746ce08-2ffb-493d-ab1a-fe3c8e45d27f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:13.793247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1746ce08-2ffb-493d-ab1a-fe3c8e45d27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:13.863394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:13.919096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 652/1324] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:14.224733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadee800-b038-4498-b807-93481eb28ec4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:14.226329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-eadee800-b038-4498-b807-93481eb28ec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:14.354841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadee800-b038-4498-b807-93481eb28ec4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:14.369051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:14.448681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919b9ec5-077b-43d0-b193-b60a60d43dc9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:14.449351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919b9ec5-077b-43d0-b193-b60a60d43dc9 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:14.449631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919b9ec5-077b-43d0-b193-b60a60d43dc9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:14.449952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919b9ec5-077b-43d0-b193-b60a60d43dc9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:14.451879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-919b9ec5-077b-43d0-b193-b60a60d43dc9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:14.583856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919b9ec5-077b-43d0-b193-b60a60d43dc9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:14.585329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919b9ec5-077b-43d0-b193-b60a60d43dc9 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:14.604187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 673/1325] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:16.680159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82941e21-70b3-4d71-aba6-d4eaf9ae4112 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:16.680850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82941e21-70b3-4d71-aba6-d4eaf9ae4112 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:16.680850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82941e21-70b3-4d71-aba6-d4eaf9ae4112 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:16.682367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 653/1326] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:16.688835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dec67d-f578-4b36-87cc-56b71c1e95ae None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:16.689481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dec67d-f578-4b36-87cc-56b71c1e95ae None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:16.690005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dec67d-f578-4b36-87cc-56b71c1e95ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:16.767042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:16.797084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-02dec67d-f578-4b36-87cc-56b71c1e95ae None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:16.798340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-02dec67d-f578-4b36-87cc-56b71c1e95ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:16.919507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 674/1327] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:17.254386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82941e21-70b3-4d71-aba6-d4eaf9ae4112 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:17.256012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-82941e21-70b3-4d71-aba6-d4eaf9ae4112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:17.417455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82941e21-70b3-4d71-aba6-d4eaf9ae4112 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:17.533050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aa33e3-a3f0-4306-a957-be7876f21fcf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:17.533690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aa33e3-a3f0-4306-a957-be7876f21fcf service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:17.534049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aa33e3-a3f0-4306-a957-be7876f21fcf service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:17.534383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aa33e3-a3f0-4306-a957-be7876f21fcf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:17.536142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e6aa33e3-a3f0-4306-a957-be7876f21fcf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:17.559777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:17.718571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6aa33e3-a3f0-4306-a957-be7876f21fcf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:17.721154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6aa33e3-a3f0-4306-a957-be7876f21fcf service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:17.756288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 654/1328] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:20.072478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a290515-4ddb-4aa8-86f9-896c4df8450e None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:20.073331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a290515-4ddb-4aa8-86f9-896c4df8450e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:20.073331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a290515-4ddb-4aa8-86f9-896c4df8450e None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:20.074623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 675/1329] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:20.085634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccadb378-86cd-446f-a65f-48aa0bb61544 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:20.086451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccadb378-86cd-446f-a65f-48aa0bb61544 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:20.086899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccadb378-86cd-446f-a65f-48aa0bb61544 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:20.175757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ccadb378-86cd-446f-a65f-48aa0bb61544 None None] MFA Rules not processed for user `1c1b7dba7cb84f5c89be5d6f9d6f60a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:20.177367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ccadb378-86cd-446f-a65f-48aa0bb61544 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:20.271546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:20.325811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 655/1330] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:20 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:20.636555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a290515-4ddb-4aa8-86f9-896c4df8450e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:20.639348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3a290515-4ddb-4aa8-86f9-896c4df8450e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:20.817232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a290515-4ddb-4aa8-86f9-896c4df8450e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:20.849601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:20.936187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c46d44-b121-4daf-bfb9-2d832d3db950 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:20.936840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c46d44-b121-4daf-bfb9-2d832d3db950 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:20.937368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c46d44-b121-4daf-bfb9-2d832d3db950 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:20.937731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c46d44-b121-4daf-bfb9-2d832d3db950 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:20.939409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d2c46d44-b121-4daf-bfb9-2d832d3db950 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:21.104321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c46d44-b121-4daf-bfb9-2d832d3db950 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:21.105894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c46d44-b121-4daf-bfb9-2d832d3db950 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:21.135088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 676/1331] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:20 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:23.634122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed3572c-8ef0-40ca-b52a-8c3862f44a7b None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:23.634786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed3572c-8ef0-40ca-b52a-8c3862f44a7b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:23.634786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed3572c-8ef0-40ca-b52a-8c3862f44a7b None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:23.636101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 656/1332] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:23.643202 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48f2852-4efa-45c6-9fb2-f4e90fbea438 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:23.643761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48f2852-4efa-45c6-9fb2-f4e90fbea438 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:23.643761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48f2852-4efa-45c6-9fb2-f4e90fbea438 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:23.740817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:23.747079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e48f2852-4efa-45c6-9fb2-f4e90fbea438 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:23.749489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e48f2852-4efa-45c6-9fb2-f4e90fbea438 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:23.875802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 677/1333] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:24.186413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed3572c-8ef0-40ca-b52a-8c3862f44a7b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:24.188890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bed3572c-8ef0-40ca-b52a-8c3862f44a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:24.373108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed3572c-8ef0-40ca-b52a-8c3862f44a7b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:24.492573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dd8b8c-d1ce-4c37-b532-d023d88cca2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:24.493678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dd8b8c-d1ce-4c37-b532-d023d88cca2d service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:24.493974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dd8b8c-d1ce-4c37-b532-d023d88cca2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:24.494219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dd8b8c-d1ce-4c37-b532-d023d88cca2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:24.496168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-67dd8b8c-d1ce-4c37-b532-d023d88cca2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:24.533885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:24.659882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dd8b8c-d1ce-4c37-b532-d023d88cca2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:24.662171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dd8b8c-d1ce-4c37-b532-d023d88cca2d service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:24.692636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 657/1334] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:26.745528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442774f-e569-4b7b-a3fa-8013a0e4a99d None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:26.746434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442774f-e569-4b7b-a3fa-8013a0e4a99d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:26.746434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442774f-e569-4b7b-a3fa-8013a0e4a99d None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:26.747932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 678/1335] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:26.756564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87886e6-bb78-4f00-b8a7-eb286cf50e86 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:26.757193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87886e6-bb78-4f00-b8a7-eb286cf50e86 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:26.757317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87886e6-bb78-4f00-b8a7-eb286cf50e86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:26.842788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e87886e6-bb78-4f00-b8a7-eb286cf50e86 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:26.844064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e87886e6-bb78-4f00-b8a7-eb286cf50e86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:26.936208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:26.962651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 658/1336] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:27.268382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0442774f-e569-4b7b-a3fa-8013a0e4a99d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:27.270207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0442774f-e569-4b7b-a3fa-8013a0e4a99d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:27.390659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0442774f-e569-4b7b-a3fa-8013a0e4a99d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:27.424753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:27.482630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9e8ae0-3244-4dbc-b1e6-fb276cd36fe8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:27.483256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e8ae0-3244-4dbc-b1e6-fb276cd36fe8 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:27.483532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e8ae0-3244-4dbc-b1e6-fb276cd36fe8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:27.483801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e8ae0-3244-4dbc-b1e6-fb276cd36fe8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:27.485516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bc9e8ae0-3244-4dbc-b1e6-fb276cd36fe8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:27.618388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9e8ae0-3244-4dbc-b1e6-fb276cd36fe8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:27.619850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9e8ae0-3244-4dbc-b1e6-fb276cd36fe8 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:27.638805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 679/1337] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:30.143330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44757634-9b07-4903-befe-5bc09db52709 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:30.143959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44757634-9b07-4903-befe-5bc09db52709 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:30.144065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44757634-9b07-4903-befe-5bc09db52709 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:30.145545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 659/1338] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:30.151501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e87338f-884f-4f71-9901-7b131ce0e9e5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:30.152197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e87338f-884f-4f71-9901-7b131ce0e9e5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:30.152197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e87338f-884f-4f71-9901-7b131ce0e9e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:30.238622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2e87338f-884f-4f71-9901-7b131ce0e9e5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:30.239853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2e87338f-884f-4f71-9901-7b131ce0e9e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:30.249344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:30.390688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 680/1339] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:30 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:14:30.691949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44757634-9b07-4903-befe-5bc09db52709 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:30.694704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-44757634-9b07-4903-befe-5bc09db52709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:30.822015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44757634-9b07-4903-befe-5bc09db52709 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:30.912637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dadcc5-865c-4340-b992-525732e40b2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:30.913358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dadcc5-865c-4340-b992-525732e40b2d service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:30.913676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dadcc5-865c-4340-b992-525732e40b2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:30.913988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dadcc5-865c-4340-b992-525732e40b2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:30.915779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-39dadcc5-865c-4340-b992-525732e40b2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:30.957992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:31.052795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dadcc5-865c-4340-b992-525732e40b2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:31.054301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dadcc5-865c-4340-b992-525732e40b2d service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:31.080158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 660/1340] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:30 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:35.721908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf486c-176e-44dd-867d-7f74140c83d1 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:35.722619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf486c-176e-44dd-867d-7f74140c83d1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:35.722619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf486c-176e-44dd-867d-7f74140c83d1 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:35.723875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 681/1341] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:35.732007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e56946-1626-4134-8803-ca34fdb11aa8 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:35.732753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e56946-1626-4134-8803-ca34fdb11aa8 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:35.732753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e56946-1626-4134-8803-ca34fdb11aa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:35.831273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-28e56946-1626-4134-8803-ca34fdb11aa8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:35.832443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-28e56946-1626-4134-8803-ca34fdb11aa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:35.984186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:35.997147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 661/1342] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:36.312503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf486c-176e-44dd-867d-7f74140c83d1 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:36.314228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cabf486c-176e-44dd-867d-7f74140c83d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:36.479097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf486c-176e-44dd-867d-7f74140c83d1 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:36.547068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:36.606380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a191667-9769-4d70-a686-df9efa24eb43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:36.607133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a191667-9769-4d70-a686-df9efa24eb43 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:36.607430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a191667-9769-4d70-a686-df9efa24eb43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:36.607818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a191667-9769-4d70-a686-df9efa24eb43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:36.610179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2a191667-9769-4d70-a686-df9efa24eb43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:36.766020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a191667-9769-4d70-a686-df9efa24eb43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:36.768505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a191667-9769-4d70-a686-df9efa24eb43 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:36.794250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 682/1343] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:14:40.025625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6232062e-1aa1-4eee-90d5-eddac80901ad None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:40.026148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6232062e-1aa1-4eee-90d5-eddac80901ad None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:40.026148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6232062e-1aa1-4eee-90d5-eddac80901ad None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:40.027443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 662/1344] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:40.036271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf92ffb-82b0-47b0-ae5f-7726322b304c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:40.036816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf92ffb-82b0-47b0-ae5f-7726322b304c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:40.037180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf92ffb-82b0-47b0-ae5f-7726322b304c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:40.120203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-cdf92ffb-82b0-47b0-ae5f-7726322b304c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:40.121462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cdf92ffb-82b0-47b0-ae5f-7726322b304c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:40.142321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:40.252305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 683/1345] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:40 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:40.567466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6232062e-1aa1-4eee-90d5-eddac80901ad None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:40.568997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6232062e-1aa1-4eee-90d5-eddac80901ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:40.714183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6232062e-1aa1-4eee-90d5-eddac80901ad None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:40.826906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279d5925-fb7b-4633-b7d8-40bdb2a64b87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:40.827850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d5925-fb7b-4633-b7d8-40bdb2a64b87 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:40.828392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d5925-fb7b-4633-b7d8-40bdb2a64b87 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:40.828789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d5925-fb7b-4633-b7d8-40bdb2a64b87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:40.831741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-279d5925-fb7b-4633-b7d8-40bdb2a64b87 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:40.899615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:41.033894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279d5925-fb7b-4633-b7d8-40bdb2a64b87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:41.035377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279d5925-fb7b-4633-b7d8-40bdb2a64b87 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:41.066124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 663/1346] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:40 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:14:46.022713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6579b-32e8-4285-81f1-b4e4bfc100bd None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:46.023364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6579b-32e8-4285-81f1-b4e4bfc100bd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:46.023544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6579b-32e8-4285-81f1-b4e4bfc100bd None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:46.025874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 684/1347] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:14:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:14:46.033450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0d896-bb54-48d3-b385-079be38a6f4a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:46.034204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0d896-bb54-48d3-b385-079be38a6f4a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:46.034204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0d896-bb54-48d3-b385-079be38a6f4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:46.126956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-04e0d896-bb54-48d3-b385-079be38a6f4a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:46.128340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-04e0d896-bb54-48d3-b385-079be38a6f4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:46.287311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:46.297788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 664/1348] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:14:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:46.682197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6579b-32e8-4285-81f1-b4e4bfc100bd None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:14:46.684761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f1e6579b-32e8-4285-81f1-b4e4bfc100bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:46.871745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6579b-32e8-4285-81f1-b4e4bfc100bd None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:14:46.937303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:46.968145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d86e2-5ef6-44f1-95fc-92b261ee09a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:14:46.968764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d86e2-5ef6-44f1-95fc-92b261ee09a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:46.969042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d86e2-5ef6-44f1-95fc-92b261ee09a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:46.969331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d86e2-5ef6-44f1-95fc-92b261ee09a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:46.971044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c28d86e2-5ef6-44f1-95fc-92b261ee09a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:47.115901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28d86e2-5ef6-44f1-95fc-92b261ee09a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:14:47.118321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28d86e2-5ef6-44f1-95fc-92b261ee09a7 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:14:47.145725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 685/1349] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:14:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:14:52.946964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1151ba60-517d-4d87-9557-d492efc78fe0 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:52.947747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1151ba60-517d-4d87-9557-d492efc78fe0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:52.951930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1151ba60-517d-4d87-9557-d492efc78fe0 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:52.955637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 665/1350] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:14:52 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:52.964741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e1c3b-b3fb-4701-9966-b3c375c0f0b5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:52.965777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e1c3b-b3fb-4701-9966-b3c375c0f0b5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:52.965972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88e1c3b-b3fb-4701-9966-b3c375c0f0b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:53.118158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a88e1c3b-b3fb-4701-9966-b3c375c0f0b5 None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:53.120212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a88e1c3b-b3fb-4701-9966-b3c375c0f0b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:53.149815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:53.355156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 686/1351] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:14:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 393 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:14:58.362464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae9939-692d-4031-babc-9c8da2f8e744 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:58.363397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae9939-692d-4031-babc-9c8da2f8e744 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:58.363845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae9939-692d-4031-babc-9c8da2f8e744 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:58.365490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 666/1352] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:14:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:14:58.372222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c872884-a9f4-443d-afa3-67fc8c860d82 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:14:58.372756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c872884-a9f4-443d-afa3-67fc8c860d82 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:14:58.373299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c872884-a9f4-443d-afa3-67fc8c860d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:14:58.489471 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5c872884-a9f4-443d-afa3-67fc8c860d82 None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:14:58.490586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5c872884-a9f4-443d-afa3-67fc8c860d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:14:58.584985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:14:58.638216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 687/1353] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:14:58 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:00.228899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6eb5c8-4a54-4fd1-8101-7f7ad82b86cd None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:00.229537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6eb5c8-4a54-4fd1-8101-7f7ad82b86cd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:00.229537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6eb5c8-4a54-4fd1-8101-7f7ad82b86cd None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:00.230790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 667/1354] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:15:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:00.236439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e30fa1-a770-47b5-9cee-22fe36b27062 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:00.236975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e30fa1-a770-47b5-9cee-22fe36b27062 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:00.237083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e30fa1-a770-47b5-9cee-22fe36b27062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:00.333809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-78e30fa1-a770-47b5-9cee-22fe36b27062 None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:00.335039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-78e30fa1-a770-47b5-9cee-22fe36b27062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:00.466685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 688/1355] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:15:00 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:15:04.107691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83076d3a-1ff2-4d2e-b23c-de1370ff4d16 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:04.108918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83076d3a-1ff2-4d2e-b23c-de1370ff4d16 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:04.108918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83076d3a-1ff2-4d2e-b23c-de1370ff4d16 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:04.109929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 668/1356] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:04.115687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d042b6-8d56-4a20-bc52-d3f89f5b9383 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:04.116212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d042b6-8d56-4a20-bc52-d3f89f5b9383 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:04.116346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d042b6-8d56-4a20-bc52-d3f89f5b9383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:04.158112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:04.207368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-71d042b6-8d56-4a20-bc52-d3f89f5b9383 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:04.209424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-71d042b6-8d56-4a20-bc52-d3f89f5b9383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:04.413715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 689/1357] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:15:04.743697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5503ec-f0ad-4343-9ae2-5bc03356af8f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:04.744720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5503ec-f0ad-4343-9ae2-5bc03356af8f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:04.745448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5503ec-f0ad-4343-9ae2-5bc03356af8f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:04.749919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 669/1358] 10.222.0.52 () {54 vars in 962 bytes} [Sat Aug 27 02:15:04 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:04.760624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c03a74-1b8e-4aae-8e3c-bc62d729b1cf None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:04.761616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c03a74-1b8e-4aae-8e3c-bc62d729b1cf None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:04.762358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c03a74-1b8e-4aae-8e3c-bc62d729b1cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:04.864386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-91c03a74-1b8e-4aae-8e3c-bc62d729b1cf None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:04.865838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-91c03a74-1b8e-4aae-8e3c-bc62d729b1cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:04.946307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:05.035258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 690/1359] 10.222.0.52 () {58 vars in 1059 bytes} [Sat Aug 27 02:15:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:05.044090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5503ec-f0ad-4343-9ae2-5bc03356af8f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:05.045618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4c5503ec-f0ad-4343-9ae2-5bc03356af8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:05.217952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5503ec-f0ad-4343-9ae2-5bc03356af8f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:05.304391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:05.356757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef29e539-1701-4f7c-a015-345a9775cdb1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:05.357613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef29e539-1701-4f7c-a015-345a9775cdb1 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:05.358119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef29e539-1701-4f7c-a015-345a9775cdb1 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:05.358530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef29e539-1701-4f7c-a015-345a9775cdb1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:05.360306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ef29e539-1701-4f7c-a015-345a9775cdb1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:05.470678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef29e539-1701-4f7c-a015-345a9775cdb1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:05.472474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef29e539-1701-4f7c-a015-345a9775cdb1 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:05.492035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 670/1360] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:07.882011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f62dd-8370-4630-9bb9-f160f4f768c4 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:07.882843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f62dd-8370-4630-9bb9-f160f4f768c4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:07.882843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f62dd-8370-4630-9bb9-f160f4f768c4 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:07.888545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 691/1361] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:07.891005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e3a33c-a89d-412f-a4b6-8d1a3e6f8358 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:07.891493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e3a33c-a89d-412f-a4b6-8d1a3e6f8358 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:07.891851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e3a33c-a89d-412f-a4b6-8d1a3e6f8358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:07.981977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-69e3a33c-a89d-412f-a4b6-8d1a3e6f8358 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:07.981977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-69e3a33c-a89d-412f-a4b6-8d1a3e6f8358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:08.024972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:08.142836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 671/1362] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:08.432487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68f62dd-8370-4630-9bb9-f160f4f768c4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:08.434265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b68f62dd-8370-4630-9bb9-f160f4f768c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:08.645750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68f62dd-8370-4630-9bb9-f160f4f768c4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:08.657227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:08.839472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46208e5-7c57-463b-8fbe-7195ec296517 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:08.840024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46208e5-7c57-463b-8fbe-7195ec296517 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:08.840772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46208e5-7c57-463b-8fbe-7195ec296517 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:08.840772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46208e5-7c57-463b-8fbe-7195ec296517 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:08.843088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d46208e5-7c57-463b-8fbe-7195ec296517 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:08.962349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46208e5-7c57-463b-8fbe-7195ec296517 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:08.964048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46208e5-7c57-463b-8fbe-7195ec296517 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:08.990010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 692/1363] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 555 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:11.337309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd8fdd1-ff93-4fb9-86c8-276cb0906317 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:11.337949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd8fdd1-ff93-4fb9-86c8-276cb0906317 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:11.337949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd8fdd1-ff93-4fb9-86c8-276cb0906317 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:11.339190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 672/1364] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:11.345028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87da98fb-2601-408c-b540-d593d44d1c42 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:11.345565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87da98fb-2601-408c-b540-d593d44d1c42 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:11.345660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87da98fb-2601-408c-b540-d593d44d1c42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:11.362792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:11.421636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-87da98fb-2601-408c-b540-d593d44d1c42 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:11.423145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-87da98fb-2601-408c-b540-d593d44d1c42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:11.530951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 693/1365] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:11.819446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f96def-6823-4979-8976-b226da02d9a6 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:11.819773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f96def-6823-4979-8976-b226da02d9a6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:11.819971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f96def-6823-4979-8976-b226da02d9a6 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:11.821624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 673/1366] 10.222.0.52 () {54 vars in 962 bytes} [Sat Aug 27 02:15:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:11.827143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e3d6d-5b06-43c8-8a0a-f0d5222a625e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:11.827526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e3d6d-5b06-43c8-8a0a-f0d5222a625e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:11.827910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e3d6d-5b06-43c8-8a0a-f0d5222a625e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:11.914902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-109e3d6d-5b06-43c8-8a0a-f0d5222a625e None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:11.916380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-109e3d6d-5b06-43c8-8a0a-f0d5222a625e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:11.957653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:12.045926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 694/1367] 10.222.0.52 () {58 vars in 1059 bytes} [Sat Aug 27 02:15:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:12.054482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f96def-6823-4979-8976-b226da02d9a6 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:12.056003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-60f96def-6823-4979-8976-b226da02d9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:12.233641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f96def-6823-4979-8976-b226da02d9a6 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:12.260230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:12.405031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbca0c4a-3119-4120-aef8-b46524c10463 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:12.405996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca0c4a-3119-4120-aef8-b46524c10463 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:12.405996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca0c4a-3119-4120-aef8-b46524c10463 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:12.406445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca0c4a-3119-4120-aef8-b46524c10463 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:12.408373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bbca0c4a-3119-4120-aef8-b46524c10463 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:12.518512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbca0c4a-3119-4120-aef8-b46524c10463 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:12.520297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbca0c4a-3119-4120-aef8-b46524c10463 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:12.546601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 674/1368] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:12.809042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403cea7-a32e-4ffb-b236-91559f55d358 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:12.809641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403cea7-a32e-4ffb-b236-91559f55d358 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:12.809641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403cea7-a32e-4ffb-b236-91559f55d358 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:12.811430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 695/1369] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:12.817586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530fe0e-1065-4823-8b11-5675d008f766 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:12.817893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530fe0e-1065-4823-8b11-5675d008f766 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:12.818138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530fe0e-1065-4823-8b11-5675d008f766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:12.854489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:12.905612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f530fe0e-1065-4823-8b11-5675d008f766 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:12.907129 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f530fe0e-1065-4823-8b11-5675d008f766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:13.029359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 675/1370] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:13.649247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221f8f14-030f-4298-aef9-11f614087937 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:13.649689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221f8f14-030f-4298-aef9-11f614087937 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:13.650084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221f8f14-030f-4298-aef9-11f614087937 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:13.652738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 696/1371] 10.222.0.52 () {54 vars in 964 bytes} [Sat Aug 27 02:15:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:13.658242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb3b42-e59c-4f38-9710-75fdb4902dc9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:13.659324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb3b42-e59c-4f38-9710-75fdb4902dc9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:13.659906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb3b42-e59c-4f38-9710-75fdb4902dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:13.754543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-90fb3b42-e59c-4f38-9710-75fdb4902dc9 None None] MFA Rules not processed for user `598bc689672741c0ac600e42cf45e39d`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:13.756045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-90fb3b42-e59c-4f38-9710-75fdb4902dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:13.809669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:13.847142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 676/1372] 10.222.0.52 () {58 vars in 1061 bytes} [Sat Aug 27 02:15:13 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:15:13.856521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221f8f14-030f-4298-aef9-11f614087937 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:13.859502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-221f8f14-030f-4298-aef9-11f614087937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:13.999280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221f8f14-030f-4298-aef9-11f614087937 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:14.054188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:14.097730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418965ef-4cc5-4ff9-ba14-1f986e3cb056 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:14.098421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418965ef-4cc5-4ff9-ba14-1f986e3cb056 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:14.098616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418965ef-4cc5-4ff9-ba14-1f986e3cb056 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:14.098891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418965ef-4cc5-4ff9-ba14-1f986e3cb056 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:14.100663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-418965ef-4cc5-4ff9-ba14-1f986e3cb056 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:14.267384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418965ef-4cc5-4ff9-ba14-1f986e3cb056 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:14.268762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418965ef-4cc5-4ff9-ba14-1f986e3cb056 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:14.312582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 697/1373] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:14.726742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa2307e-5b6f-4bb0-968e-4378cc226b58 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:14.727228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa2307e-5b6f-4bb0-968e-4378cc226b58 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:14.727324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa2307e-5b6f-4bb0-968e-4378cc226b58 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:14.728812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 677/1374] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:14.734658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5a71a-c902-4149-bfe6-6e094116aac8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:14.734941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5a71a-c902-4149-bfe6-6e094116aac8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:14.735212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5a71a-c902-4149-bfe6-6e094116aac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:14.805709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-86d5a71a-c902-4149-bfe6-6e094116aac8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:14.806963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-86d5a71a-c902-4149-bfe6-6e094116aac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:14.832099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:14.927730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 698/1375] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:15.225983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa2307e-5b6f-4bb0-968e-4378cc226b58 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:15.227659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5aa2307e-5b6f-4bb0-968e-4378cc226b58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:15.403005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa2307e-5b6f-4bb0-968e-4378cc226b58 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:15.461310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:15.530324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a85723-f479-436e-8fa9-38ad1c41b355 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:15.531170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a85723-f479-436e-8fa9-38ad1c41b355 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:15.531431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a85723-f479-436e-8fa9-38ad1c41b355 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:15.531699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a85723-f479-436e-8fa9-38ad1c41b355 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:15.533851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-59a85723-f479-436e-8fa9-38ad1c41b355 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:15.631214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a85723-f479-436e-8fa9-38ad1c41b355 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:15.633875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a85723-f479-436e-8fa9-38ad1c41b355 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:15.666281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 678/1376] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:18.022411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d9883-4d4c-4e8a-b470-6743f13f579a None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:18.022941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d9883-4d4c-4e8a-b470-6743f13f579a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:18.023334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d9883-4d4c-4e8a-b470-6743f13f579a None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:18.025684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 699/1377] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:18.033158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabcd981-5945-4c16-ac1f-d48218805009 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:18.033670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabcd981-5945-4c16-ac1f-d48218805009 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:18.033787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabcd981-5945-4c16-ac1f-d48218805009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:18.113575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-aabcd981-5945-4c16-ac1f-d48218805009 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:18.114807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-aabcd981-5945-4c16-ac1f-d48218805009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:18.122556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:18.269354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 679/1378] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:18.602093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3d9883-4d4c-4e8a-b470-6743f13f579a None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:18.604031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9e3d9883-4d4c-4e8a-b470-6743f13f579a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:18.794439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3d9883-4d4c-4e8a-b470-6743f13f579a None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:18.798630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:18.928015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7262acf-b7a1-4b0f-ac81-74e7dfee1a43 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:18.928625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7262acf-b7a1-4b0f-ac81-74e7dfee1a43 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:18.928901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7262acf-b7a1-4b0f-ac81-74e7dfee1a43 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:18.930527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7262acf-b7a1-4b0f-ac81-74e7dfee1a43 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:18.931737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f7262acf-b7a1-4b0f-ac81-74e7dfee1a43 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:19.066584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7262acf-b7a1-4b0f-ac81-74e7dfee1a43 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:19.068632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7262acf-b7a1-4b0f-ac81-74e7dfee1a43 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:19.091772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 700/1379] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:21.180788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289fad8-bcff-47dc-8d04-16ae14aa67dc None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:21.181431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289fad8-bcff-47dc-8d04-16ae14aa67dc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:21.181431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289fad8-bcff-47dc-8d04-16ae14aa67dc None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:21.182759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 680/1380] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:21.188465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aefba38-ec4d-4f2e-bdc2-d7fd28a2ecd2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:21.188956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aefba38-ec4d-4f2e-bdc2-d7fd28a2ecd2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:21.188956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aefba38-ec4d-4f2e-bdc2-d7fd28a2ecd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:21.201148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:21.259505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-8aefba38-ec4d-4f2e-bdc2-d7fd28a2ecd2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:21.261097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8aefba38-ec4d-4f2e-bdc2-d7fd28a2ecd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:21.381798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 701/1381] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:15:21.675310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6289fad8-bcff-47dc-8d04-16ae14aa67dc None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:21.677143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6289fad8-bcff-47dc-8d04-16ae14aa67dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:21.834616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:21.852225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6289fad8-bcff-47dc-8d04-16ae14aa67dc None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:21.982024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5f9357-0485-457c-be2c-7f20cc36d45e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:21.982555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5f9357-0485-457c-be2c-7f20cc36d45e service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:21.982888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5f9357-0485-457c-be2c-7f20cc36d45e service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:21.983161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5f9357-0485-457c-be2c-7f20cc36d45e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:21.984819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0d5f9357-0485-457c-be2c-7f20cc36d45e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:22.112384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5f9357-0485-457c-be2c-7f20cc36d45e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:22.114145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5f9357-0485-457c-be2c-7f20cc36d45e service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:22.125517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:22.136771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 681/1382] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:24.241518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a1c3f2-bd4b-47be-99ea-138025ece306 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:24.242326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a1c3f2-bd4b-47be-99ea-138025ece306 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:24.242326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a1c3f2-bd4b-47be-99ea-138025ece306 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:24.244143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 702/1383] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:15:24.255744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cad43c-5c70-46a9-ad0d-5db8e40a27fd None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:24.256440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cad43c-5c70-46a9-ad0d-5db8e40a27fd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:24.256440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cad43c-5c70-46a9-ad0d-5db8e40a27fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:24.353367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-d3cad43c-5c70-46a9-ad0d-5db8e40a27fd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:24.355306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d3cad43c-5c70-46a9-ad0d-5db8e40a27fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:24.513330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 682/1384] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:15:24.820311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a1c3f2-bd4b-47be-99ea-138025ece306 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:24.823246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-08a1c3f2-bd4b-47be-99ea-138025ece306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:24.977701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:25.062129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a1c3f2-bd4b-47be-99ea-138025ece306 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:25.223457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbc0bcd-2d43-4f71-9e97-71a86b71d3b2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:25.224478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbc0bcd-2d43-4f71-9e97-71a86b71d3b2 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:25.224951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbc0bcd-2d43-4f71-9e97-71a86b71d3b2 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:25.225456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbc0bcd-2d43-4f71-9e97-71a86b71d3b2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:25.228398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-acbc0bcd-2d43-4f71-9e97-71a86b71d3b2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:25.292193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:25.345406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbc0bcd-2d43-4f71-9e97-71a86b71d3b2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:25.347508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbc0bcd-2d43-4f71-9e97-71a86b71d3b2 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:25.372737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 703/1385] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 553 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:27.422502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde45a2a-b2f3-4836-9285-29f8c4082634 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:27.423047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde45a2a-b2f3-4836-9285-29f8c4082634 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:27.423162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde45a2a-b2f3-4836-9285-29f8c4082634 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:27.424567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 683/1386] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:27.431054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3f264f-2ef6-4339-bbc5-7577b6c4a63b None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:27.431561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3f264f-2ef6-4339-bbc5-7577b6c4a63b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:27.431651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3f264f-2ef6-4339-bbc5-7577b6c4a63b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:27.503350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3b3f264f-2ef6-4339-bbc5-7577b6c4a63b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:27.504736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3b3f264f-2ef6-4339-bbc5-7577b6c4a63b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:27.634304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 704/1387] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:27.946372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde45a2a-b2f3-4836-9285-29f8c4082634 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:27.949329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dde45a2a-b2f3-4836-9285-29f8c4082634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:28.051757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:28.134799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde45a2a-b2f3-4836-9285-29f8c4082634 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:28.269724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6a02a3-e235-45da-b075-d1e53ab65041 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:28.270260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a02a3-e235-45da-b075-d1e53ab65041 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:28.270561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a02a3-e235-45da-b075-d1e53ab65041 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:28.270867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a02a3-e235-45da-b075-d1e53ab65041 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:28.272426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4e6a02a3-e235-45da-b075-d1e53ab65041 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:28.316032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:28.377185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a02a3-e235-45da-b075-d1e53ab65041 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:28.379343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a02a3-e235-45da-b075-d1e53ab65041 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:28.398419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 684/1388] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:30.427923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bf65c-d7b1-44e4-941d-96efe17d0976 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:30.428491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bf65c-d7b1-44e4-941d-96efe17d0976 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:30.428639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bf65c-d7b1-44e4-941d-96efe17d0976 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:30.430109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 705/1389] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:30.439225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7984e56e-969b-4c47-8582-7a059084941c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:30.439225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7984e56e-969b-4c47-8582-7a059084941c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:30.439225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7984e56e-969b-4c47-8582-7a059084941c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:30.458502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb11f18-2145-4817-801a-cb2fb6eef033 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:30.458766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb11f18-2145-4817-801a-cb2fb6eef033 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:30.459151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb11f18-2145-4817-801a-cb2fb6eef033 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:30.460523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 706/1390] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:30.466369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd65cdc-4638-42a4-a64f-0be69b7f31f7 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:30.466675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd65cdc-4638-42a4-a64f-0be69b7f31f7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:30.466948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd65cdc-4638-42a4-a64f-0be69b7f31f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:30.503531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:30.518996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7984e56e-969b-4c47-8582-7a059084941c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:30.520217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7984e56e-969b-4c47-8582-7a059084941c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:30.545618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5dd65cdc-4638-42a4-a64f-0be69b7f31f7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:30.547494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5dd65cdc-4638-42a4-a64f-0be69b7f31f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:30.642852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 685/1391] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:30.669913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 707/1392] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:15:30.963697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7984e56e-969b-4c47-8582-7a059084941c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:30.964963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7984e56e-969b-4c47-8582-7a059084941c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:30.980301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:31.006046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad6a3a-c8ae-4929-823a-9d6316b8f4a1 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:31.006987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad6a3a-c8ae-4929-823a-9d6316b8f4a1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:31.006987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad6a3a-c8ae-4929-823a-9d6316b8f4a1 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:31.008618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 708/1393] 10.222.0.52 () {54 vars in 967 bytes} [Sat Aug 27 02:15:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:31.014971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d007e9a6-0ea8-479a-abe9-4812011944ee None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:31.015230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d007e9a6-0ea8-479a-abe9-4812011944ee None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:31.015504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d007e9a6-0ea8-479a-abe9-4812011944ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:31.106411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7984e56e-969b-4c47-8582-7a059084941c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:31.119189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d007e9a6-0ea8-479a-abe9-4812011944ee None None] MFA Rules not processed for user `ab71b242a16641429c29dc8aa67867e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:31.120542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d007e9a6-0ea8-479a-abe9-4812011944ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:31.170862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:31.200976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 709/1394] 10.222.0.52 () {58 vars in 1064 bytes} [Sat Aug 27 02:15:31 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:31.210190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d007e9a6-0ea8-479a-abe9-4812011944ee None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:31.218249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d007e9a6-0ea8-479a-abe9-4812011944ee None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:31.231590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe872e1-8c91-4e62-9035-552a22b07247 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:31.232206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe872e1-8c91-4e62-9035-552a22b07247 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:31.232481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe872e1-8c91-4e62-9035-552a22b07247 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:31.232749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe872e1-8c91-4e62-9035-552a22b07247 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:31.235121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dfe872e1-8c91-4e62-9035-552a22b07247 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:31.243715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:31.312518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0432ed-ba3f-45ac-9ed4-c02804e80404 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:31.313155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0432ed-ba3f-45ac-9ed4-c02804e80404 service glance] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:31.313431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0432ed-ba3f-45ac-9ed4-c02804e80404 service glance] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:31.313699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0432ed-ba3f-45ac-9ed4-c02804e80404 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:31.315361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bd0432ed-ba3f-45ac-9ed4-c02804e80404 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:31.360376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe872e1-8c91-4e62-9035-552a22b07247 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:31.362467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe872e1-8c91-4e62-9035-552a22b07247 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:31.384986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 686/1395] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:15:31.469594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0432ed-ba3f-45ac-9ed4-c02804e80404 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:31.472720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0432ed-ba3f-45ac-9ed4-c02804e80404 service glance] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:31.494699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:31.501403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 710/1396] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:32.337946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d41f686-c4e4-4b20-b2db-a1cddfe371d6 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:32.339881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d41f686-c4e4-4b20-b2db-a1cddfe371d6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:32.339881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d41f686-c4e4-4b20-b2db-a1cddfe371d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:32.440258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2d41f686-c4e4-4b20-b2db-a1cddfe371d6 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:32.441526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2d41f686-c4e4-4b20-b2db-a1cddfe371d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:32.539652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:32.544301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 687/1397] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:15:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:32.601075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f962a29-c3e1-4f39-af03-f85529d45b58 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:32.601645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f962a29-c3e1-4f39-af03-f85529d45b58 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:32.601761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f962a29-c3e1-4f39-af03-f85529d45b58 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:32.603313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 711/1398] 10.222.0.52 () {54 vars in 943 bytes} [Sat Aug 27 02:15:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:15:32.610133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aec91a-da68-43d2-8535-8e2c60094df5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:32.610412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aec91a-da68-43d2-8535-8e2c60094df5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:32.610681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8aec91a-da68-43d2-8535-8e2c60094df5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:32.711540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b8aec91a-da68-43d2-8535-8e2c60094df5 None None] MFA Rules not processed for user `cfc9a334f972474aa19aa3beae51560f`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:32.712929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b8aec91a-da68-43d2-8535-8e2c60094df5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:32.844882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 688/1399] 10.222.0.52 () {58 vars in 1040 bytes} [Sat Aug 27 02:15:32 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:32.857425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f962a29-c3e1-4f39-af03-f85529d45b58 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:32.859121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3f962a29-c3e1-4f39-af03-f85529d45b58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:33.064919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f962a29-c3e1-4f39-af03-f85529d45b58 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:33.158825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:33.196407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdc6314-e50c-4fa8-9bb5-90e6c5cf77bf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:33.197647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdc6314-e50c-4fa8-9bb5-90e6c5cf77bf service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:33.198582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdc6314-e50c-4fa8-9bb5-90e6c5cf77bf service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:33.198779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdc6314-e50c-4fa8-9bb5-90e6c5cf77bf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:33.201284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5fdc6314-e50c-4fa8-9bb5-90e6c5cf77bf service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:33.339244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdc6314-e50c-4fa8-9bb5-90e6c5cf77bf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:33.341044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdc6314-e50c-4fa8-9bb5-90e6c5cf77bf service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:33.350749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 712/1400] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:15:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:15:33.427420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670bc751-45ee-45d4-b8b8-f61f6861342a None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:33.427951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670bc751-45ee-45d4-b8b8-f61f6861342a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:33.427951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670bc751-45ee-45d4-b8b8-f61f6861342a None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:33.429547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 689/1401] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:33.436245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d178a0b0-c90d-48b4-a160-7a6a0f89795c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:33.436515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d178a0b0-c90d-48b4-a160-7a6a0f89795c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:33.436791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d178a0b0-c90d-48b4-a160-7a6a0f89795c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:33.546960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d178a0b0-c90d-48b4-a160-7a6a0f89795c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:33.548632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d178a0b0-c90d-48b4-a160-7a6a0f89795c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:33.644899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:33.684131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 713/1402] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:33.984383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670bc751-45ee-45d4-b8b8-f61f6861342a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:33.986568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-670bc751-45ee-45d4-b8b8-f61f6861342a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:34.074633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:34.215644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670bc751-45ee-45d4-b8b8-f61f6861342a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:34.407789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6511d-e23b-45a4-93fe-2f1f07ab4a02 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:34.408403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6511d-e23b-45a4-93fe-2f1f07ab4a02 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:34.408950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6511d-e23b-45a4-93fe-2f1f07ab4a02 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:34.408950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6511d-e23b-45a4-93fe-2f1f07ab4a02 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:34.410743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1aa6511d-e23b-45a4-93fe-2f1f07ab4a02 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:34.433540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:34.513391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6511d-e23b-45a4-93fe-2f1f07ab4a02 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:34.515348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6511d-e23b-45a4-93fe-2f1f07ab4a02 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:34.538665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 690/1403] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 554 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:36.603167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1020f2e6-c86f-4a01-be0a-4095d0327458 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:36.603729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1020f2e6-c86f-4a01-be0a-4095d0327458 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:36.604041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1020f2e6-c86f-4a01-be0a-4095d0327458 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:36.607110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 714/1404] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:36.620664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8f0a46-63d4-4f58-a04b-f583195d625e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:36.621301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8f0a46-63d4-4f58-a04b-f583195d625e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:36.621623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8f0a46-63d4-4f58-a04b-f583195d625e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:36.706194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-dd8f0a46-63d4-4f58-a04b-f583195d625e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:36.707306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dd8f0a46-63d4-4f58-a04b-f583195d625e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:36.836824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:36.846663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 691/1405] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:37.167913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1020f2e6-c86f-4a01-be0a-4095d0327458 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:37.169394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1020f2e6-c86f-4a01-be0a-4095d0327458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:37.389050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1020f2e6-c86f-4a01-be0a-4095d0327458 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:37.479424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:37.584467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd9e80-3f14-4297-b5e1-d5f4880de6df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:37.585082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd9e80-3f14-4297-b5e1-d5f4880de6df service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:37.585387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd9e80-3f14-4297-b5e1-d5f4880de6df service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:37.585688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd9e80-3f14-4297-b5e1-d5f4880de6df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:37.587367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-18bd9e80-3f14-4297-b5e1-d5f4880de6df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:37.704264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd9e80-3f14-4297-b5e1-d5f4880de6df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:37.707639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd9e80-3f14-4297-b5e1-d5f4880de6df service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:37.728203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 715/1406] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 561 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:39.957256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e71528-0d07-4a62-912c-a593abf8fe11 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:39.958041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e71528-0d07-4a62-912c-a593abf8fe11 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:39.958041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e71528-0d07-4a62-912c-a593abf8fe11 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:39.964412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 692/1407] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:15:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:39.966022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba82ac5-7311-41a7-aff2-323d90117393 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:39.966613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba82ac5-7311-41a7-aff2-323d90117393 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:39.966613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba82ac5-7311-41a7-aff2-323d90117393 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:40.056923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5ba82ac5-7311-41a7-aff2-323d90117393 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:40.058802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5ba82ac5-7311-41a7-aff2-323d90117393 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:40.081253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:40.234867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 716/1408] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:15:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:40.586259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e71528-0d07-4a62-912c-a593abf8fe11 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:40.587802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f4e71528-0d07-4a62-912c-a593abf8fe11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:40.853099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e71528-0d07-4a62-912c-a593abf8fe11 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:40.977673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:41.048360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3764e004-c376-4705-b681-1e6577df915d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:41.049038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3764e004-c376-4705-b681-1e6577df915d service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:41.049342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3764e004-c376-4705-b681-1e6577df915d service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:41.049603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3764e004-c376-4705-b681-1e6577df915d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:41.051721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3764e004-c376-4705-b681-1e6577df915d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:41.316265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3764e004-c376-4705-b681-1e6577df915d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:41.319769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3764e004-c376-4705-b681-1e6577df915d service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:41.351546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 693/1409] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 766 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:15:50.225508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=38 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:51.345481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=64 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:56.125057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7879b4-9196-4196-b375-dcc8a97de972 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:56.125877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7879b4-9196-4196-b375-dcc8a97de972 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:56.125877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7879b4-9196-4196-b375-dcc8a97de972 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:56.126981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 717/1410] 10.222.0.52 () {54 vars in 918 bytes} [Sat Aug 27 02:15:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:56.132566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3036f87b-6d37-4d00-b3db-accb76900588 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:56.133033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3036f87b-6d37-4d00-b3db-accb76900588 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:56.133033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3036f87b-6d37-4d00-b3db-accb76900588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:56.217023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3036f87b-6d37-4d00-b3db-accb76900588 None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:56.218283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3036f87b-6d37-4d00-b3db-accb76900588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:56.334553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 694/1411] 10.222.0.52 () {58 vars in 1015 bytes} [Sat Aug 27 02:15:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:56.349782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f780148-e952-4145-8fba-326c9adb96ba None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:56.350063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f780148-e952-4145-8fba-326c9adb96ba None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:56.350335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f780148-e952-4145-8fba-326c9adb96ba None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:56.351810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 718/1412] 10.222.0.52 () {54 vars in 918 bytes} [Sat Aug 27 02:15:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:56.357282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cab60e8-4131-4cb2-b094-72483a03a0e4 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:56.357603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cab60e8-4131-4cb2-b094-72483a03a0e4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:56.357920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cab60e8-4131-4cb2-b094-72483a03a0e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:56.437149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:56.440280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3cab60e8-4131-4cb2-b094-72483a03a0e4 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:56.442247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3cab60e8-4131-4cb2-b094-72483a03a0e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:56.569167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 695/1413] 10.222.0.52 () {58 vars in 1015 bytes} [Sat Aug 27 02:15:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:59.024663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aeabd5-82ab-4a3f-8fea-83d0617e53a8 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:59.025652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aeabd5-82ab-4a3f-8fea-83d0617e53a8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:59.025652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aeabd5-82ab-4a3f-8fea-83d0617e53a8 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:59.027743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 719/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:15:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:15:59.035133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115be0bc-a036-477a-88bb-57b0ba2be7da None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:59.035766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115be0bc-a036-477a-88bb-57b0ba2be7da None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:59.035766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115be0bc-a036-477a-88bb-57b0ba2be7da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:59.114906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-115be0bc-a036-477a-88bb-57b0ba2be7da None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:15:59.116982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-115be0bc-a036-477a-88bb-57b0ba2be7da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:59.181570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:59.240712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 696/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:15:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:15:59.533866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aeabd5-82ab-4a3f-8fea-83d0617e53a8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:15:59.535569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f1aeabd5-82ab-4a3f-8fea-83d0617e53a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:59.711470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aeabd5-82ab-4a3f-8fea-83d0617e53a8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:15:59.804300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:15:59.857144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c985cdc3-f454-4541-a563-01dc9956adb4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:15:59.857899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c985cdc3-f454-4541-a563-01dc9956adb4 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:15:59.858361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c985cdc3-f454-4541-a563-01dc9956adb4 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:15:59.858658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c985cdc3-f454-4541-a563-01dc9956adb4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:15:59.861133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c985cdc3-f454-4541-a563-01dc9956adb4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:15:59.972260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c985cdc3-f454-4541-a563-01dc9956adb4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:15:59.973961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c985cdc3-f454-4541-a563-01dc9956adb4 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:15:59.994448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 720/1416] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:15:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:00.927452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf80131-aa60-4ce3-905d-50a8ad75d0dd None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:00.928007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf80131-aa60-4ce3-905d-50a8ad75d0dd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:00.928007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf80131-aa60-4ce3-905d-50a8ad75d0dd None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:00.929437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 697/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:00.936708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1055229-e250-465d-a94e-6df2ecb3e345 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:00.936708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1055229-e250-465d-a94e-6df2ecb3e345 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:00.936708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1055229-e250-465d-a94e-6df2ecb3e345 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:01.028570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d1055229-e250-465d-a94e-6df2ecb3e345 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:01.029738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d1055229-e250-465d-a94e-6df2ecb3e345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:01.063817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:01.153870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 721/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:16:01.326206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf80131-aa60-4ce3-905d-50a8ad75d0dd None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:01.327797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3cf80131-aa60-4ce3-905d-50a8ad75d0dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:01.504575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf80131-aa60-4ce3-905d-50a8ad75d0dd None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:01.549899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:01.632203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f5120-94c2-46fb-9763-790f62c5abb0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:01.632692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f5120-94c2-46fb-9763-790f62c5abb0 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:01.632980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f5120-94c2-46fb-9763-790f62c5abb0 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:01.633266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f5120-94c2-46fb-9763-790f62c5abb0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:01.634937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b35f5120-94c2-46fb-9763-790f62c5abb0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:01.738746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f5120-94c2-46fb-9763-790f62c5abb0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:01.740371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f5120-94c2-46fb-9763-790f62c5abb0 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:01.759663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 698/1419] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:02.674408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eafefd-0521-4eea-a554-cbfe6e0d41c5 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:02.675019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eafefd-0521-4eea-a554-cbfe6e0d41c5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:02.675019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eafefd-0521-4eea-a554-cbfe6e0d41c5 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:02.676376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 722/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:02.682306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4224fe-e9a0-4052-bc34-bacc6e205f6e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:02.682800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4224fe-e9a0-4052-bc34-bacc6e205f6e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:02.682800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4224fe-e9a0-4052-bc34-bacc6e205f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:02.742974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:02.767494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ca4224fe-e9a0-4052-bc34-bacc6e205f6e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:02.768991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ca4224fe-e9a0-4052-bc34-bacc6e205f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:02.873440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 699/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:03.028963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eafefd-0521-4eea-a554-cbfe6e0d41c5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:03.030564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-01eafefd-0521-4eea-a554-cbfe6e0d41c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:03.204439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eafefd-0521-4eea-a554-cbfe6e0d41c5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:03.216014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:03.333395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c862093-4949-41be-8501-4d1f1726579c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:03.334096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c862093-4949-41be-8501-4d1f1726579c service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:03.334396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c862093-4949-41be-8501-4d1f1726579c service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:03.334719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c862093-4949-41be-8501-4d1f1726579c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:03.336483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2c862093-4949-41be-8501-4d1f1726579c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:03.447270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c862093-4949-41be-8501-4d1f1726579c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:03.449529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c862093-4949-41be-8501-4d1f1726579c service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:03.471195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 723/1422] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:04.389416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f86a69-38e8-4e2b-98ba-cbd61031935a None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:04.390075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f86a69-38e8-4e2b-98ba-cbd61031935a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:04.390075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f86a69-38e8-4e2b-98ba-cbd61031935a None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:04.392030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 700/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:04.398920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f01e3-9d19-4ff2-a76f-170d94838d68 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:04.399450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f01e3-9d19-4ff2-a76f-170d94838d68 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:04.399450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f01e3-9d19-4ff2-a76f-170d94838d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:04.422462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:04.488330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-775f01e3-9d19-4ff2-a76f-170d94838d68 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:04.489606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-775f01e3-9d19-4ff2-a76f-170d94838d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:04.594171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 724/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:04.752339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f86a69-38e8-4e2b-98ba-cbd61031935a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:04.753889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e0f86a69-38e8-4e2b-98ba-cbd61031935a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:04.887197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:04.927915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f86a69-38e8-4e2b-98ba-cbd61031935a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:05.057852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffa4ba3-4b47-453a-bf7f-18c371731b46 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:05.058593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa4ba3-4b47-453a-bf7f-18c371731b46 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:05.059436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa4ba3-4b47-453a-bf7f-18c371731b46 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:05.059704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa4ba3-4b47-453a-bf7f-18c371731b46 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:05.061394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dffa4ba3-4b47-453a-bf7f-18c371731b46 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:05.150423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:05.165834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffa4ba3-4b47-453a-bf7f-18c371731b46 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:05.167442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffa4ba3-4b47-453a-bf7f-18c371731b46 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:05.188736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 701/1425] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:06.076053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db33f6c-9380-4b48-8aa7-1fe2619a8136 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:06.076710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db33f6c-9380-4b48-8aa7-1fe2619a8136 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:06.076991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db33f6c-9380-4b48-8aa7-1fe2619a8136 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:06.078758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 725/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:06.085018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257525b-d463-41ed-9f7d-87a176385002 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:06.085550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257525b-d463-41ed-9f7d-87a176385002 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:06.085638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257525b-d463-41ed-9f7d-87a176385002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:06.171506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8257525b-d463-41ed-9f7d-87a176385002 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:06.172726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8257525b-d463-41ed-9f7d-87a176385002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:06.280759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 702/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:06.438761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db33f6c-9380-4b48-8aa7-1fe2619a8136 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:06.440295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2db33f6c-9380-4b48-8aa7-1fe2619a8136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:06.543726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:06.611147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db33f6c-9380-4b48-8aa7-1fe2619a8136 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:06.735736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e7a7e9-3019-4680-9a9c-d49adc82f187 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:06.736362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e7a7e9-3019-4680-9a9c-d49adc82f187 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:06.736638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e7a7e9-3019-4680-9a9c-d49adc82f187 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:06.736906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e7a7e9-3019-4680-9a9c-d49adc82f187 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:06.738644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-18e7a7e9-3019-4680-9a9c-d49adc82f187 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:06.793533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:06.847800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e7a7e9-3019-4680-9a9c-d49adc82f187 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:06.849510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e7a7e9-3019-4680-9a9c-d49adc82f187 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:06.871120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 726/1428] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:07.694328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969dc93e-4fec-42ec-b193-c3d36e794a67 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:07.695213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969dc93e-4fec-42ec-b193-c3d36e794a67 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:07.695213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969dc93e-4fec-42ec-b193-c3d36e794a67 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:07.696657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 703/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:07.704151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7483efb-3b6d-4aa4-8fb6-7c6092176d17 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:07.706244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7483efb-3b6d-4aa4-8fb6-7c6092176d17 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:07.706244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7483efb-3b6d-4aa4-8fb6-7c6092176d17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:07.807712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c7483efb-3b6d-4aa4-8fb6-7c6092176d17 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:07.809774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c7483efb-3b6d-4aa4-8fb6-7c6092176d17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:07.937565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 727/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:08.103568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969dc93e-4fec-42ec-b193-c3d36e794a67 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:08.105089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-969dc93e-4fec-42ec-b193-c3d36e794a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:08.165212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:08.275589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969dc93e-4fec-42ec-b193-c3d36e794a67 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:08.407802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990d3e9d-8b49-48e0-b005-230ea97c1271 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:08.408696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990d3e9d-8b49-48e0-b005-230ea97c1271 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:08.409103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990d3e9d-8b49-48e0-b005-230ea97c1271 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:08.409551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990d3e9d-8b49-48e0-b005-230ea97c1271 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:08.412091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-990d3e9d-8b49-48e0-b005-230ea97c1271 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:08.424219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:08.525525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990d3e9d-8b49-48e0-b005-230ea97c1271 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:08.527285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990d3e9d-8b49-48e0-b005-230ea97c1271 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:08.547434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 704/1431] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:09.364714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d20bd6f-e3c1-4a95-9772-aa48c5a38dbe None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:09.365665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d20bd6f-e3c1-4a95-9772-aa48c5a38dbe None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:09.365665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d20bd6f-e3c1-4a95-9772-aa48c5a38dbe None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:09.366600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 728/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:09.372250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892e773b-a9ef-451d-9e56-c8131afdad6c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:09.372821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892e773b-a9ef-451d-9e56-c8131afdad6c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:09.372821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892e773b-a9ef-451d-9e56-c8131afdad6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:09.462916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-892e773b-a9ef-451d-9e56-c8131afdad6c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:09.464144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-892e773b-a9ef-451d-9e56-c8131afdad6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:09.542469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:09.575195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 705/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:09.733182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d20bd6f-e3c1-4a95-9772-aa48c5a38dbe None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:09.734931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9d20bd6f-e3c1-4a95-9772-aa48c5a38dbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:09.756456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:09.899896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d20bd6f-e3c1-4a95-9772-aa48c5a38dbe None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:10.004973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:10.033818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d666af6-2afa-48d6-85d6-d2d114747aa7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:10.034445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d666af6-2afa-48d6-85d6-d2d114747aa7 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:10.034721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d666af6-2afa-48d6-85d6-d2d114747aa7 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:10.034990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d666af6-2afa-48d6-85d6-d2d114747aa7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:10.036975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4d666af6-2afa-48d6-85d6-d2d114747aa7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:10.133053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d666af6-2afa-48d6-85d6-d2d114747aa7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:10.134697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d666af6-2afa-48d6-85d6-d2d114747aa7 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:10.154608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 729/1434] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:11.003025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a0fa2-b89b-41b0-a210-d8375922241b None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:11.003563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a0fa2-b89b-41b0-a210-d8375922241b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:11.003563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a0fa2-b89b-41b0-a210-d8375922241b None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:11.005300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 706/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:11.011279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb8249-48c7-4843-b605-a7e3c65bb1ef None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:11.012087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb8249-48c7-4843-b605-a7e3c65bb1ef None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:11.012303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb8249-48c7-4843-b605-a7e3c65bb1ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:11.115735 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-7edb8249-48c7-4843-b605-a7e3c65bb1ef None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:11.117228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7edb8249-48c7-4843-b605-a7e3c65bb1ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:11.164348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:11.238676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 730/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:11.392742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308a0fa2-b89b-41b0-a210-d8375922241b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:11.394381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-308a0fa2-b89b-41b0-a210-d8375922241b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:11.563811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308a0fa2-b89b-41b0-a210-d8375922241b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:11.627586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:11.692314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc63a5fd-ec7c-4b62-8365-c79d719d12f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:11.692842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63a5fd-ec7c-4b62-8365-c79d719d12f4 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:11.693112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63a5fd-ec7c-4b62-8365-c79d719d12f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:11.693480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63a5fd-ec7c-4b62-8365-c79d719d12f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:11.695099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-cc63a5fd-ec7c-4b62-8365-c79d719d12f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:11.795911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc63a5fd-ec7c-4b62-8365-c79d719d12f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:11.797558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc63a5fd-ec7c-4b62-8365-c79d719d12f4 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:11.817218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 707/1437] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:12.675244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f10568-c31f-4d80-ac4b-c7f8f1d9f7aa None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:12.675876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f10568-c31f-4d80-ac4b-c7f8f1d9f7aa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:12.675876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f10568-c31f-4d80-ac4b-c7f8f1d9f7aa None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:12.677143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 731/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:16:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:12.684580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aa3e47-900d-47ed-ba15-2fb0ab87d0a3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:12.685081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aa3e47-900d-47ed-ba15-2fb0ab87d0a3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:12.685238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aa3e47-900d-47ed-ba15-2fb0ab87d0a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:12.777645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-52aa3e47-900d-47ed-ba15-2fb0ab87d0a3 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:12.778961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-52aa3e47-900d-47ed-ba15-2fb0ab87d0a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:12.781186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:12.893099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 708/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:16:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:13.055942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f10568-c31f-4d80-ac4b-c7f8f1d9f7aa None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:13.057411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d6f10568-c31f-4d80-ac4b-c7f8f1d9f7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:13.226190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f10568-c31f-4d80-ac4b-c7f8f1d9f7aa None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:13.261018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:13.359444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8134a6ff-4144-4bb9-99a8-0b3ed4bd15fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:13.360068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134a6ff-4144-4bb9-99a8-0b3ed4bd15fc service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:13.360346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134a6ff-4144-4bb9-99a8-0b3ed4bd15fc service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:13.360612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134a6ff-4144-4bb9-99a8-0b3ed4bd15fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:13.362337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8134a6ff-4144-4bb9-99a8-0b3ed4bd15fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:13.467128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8134a6ff-4144-4bb9-99a8-0b3ed4bd15fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:13.468718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8134a6ff-4144-4bb9-99a8-0b3ed4bd15fc service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:13.488066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 732/1440] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:15.346947 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0439c96-22a4-4d3f-b611-26868b812969 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:15.347551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0439c96-22a4-4d3f-b611-26868b812969 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:15.347551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0439c96-22a4-4d3f-b611-26868b812969 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:15.349044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 709/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:15.354776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed0e08-da9b-401a-bd9f-5da7082d98f1 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:15.355270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed0e08-da9b-401a-bd9f-5da7082d98f1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:15.355389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed0e08-da9b-401a-bd9f-5da7082d98f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:15.403057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:15.431888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f9ed0e08-da9b-401a-bd9f-5da7082d98f1 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:15.433092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f9ed0e08-da9b-401a-bd9f-5da7082d98f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:15.541957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 733/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:15.823863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0439c96-22a4-4d3f-b611-26868b812969 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:15.826426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b0439c96-22a4-4d3f-b611-26868b812969 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:15.952644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0439c96-22a4-4d3f-b611-26868b812969 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:15.996761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:16.046602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bdd9a-099f-4114-bdf8-6e8b76f5ecaf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:16.047231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bdd9a-099f-4114-bdf8-6e8b76f5ecaf service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:16.047498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bdd9a-099f-4114-bdf8-6e8b76f5ecaf service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:16.047760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bdd9a-099f-4114-bdf8-6e8b76f5ecaf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:16.049423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-281bdd9a-099f-4114-bdf8-6e8b76f5ecaf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:16.176024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bdd9a-099f-4114-bdf8-6e8b76f5ecaf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:16.177506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bdd9a-099f-4114-bdf8-6e8b76f5ecaf service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:16.200447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 710/1443] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:18.389275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adbae41-8796-4770-8da1-a462d28213d2 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:18.389890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adbae41-8796-4770-8da1-a462d28213d2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:18.389890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adbae41-8796-4770-8da1-a462d28213d2 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:18.391171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 734/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:18.396595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf6de8-aff9-473e-a41f-4f8bde47a0e6 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:18.397043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf6de8-aff9-473e-a41f-4f8bde47a0e6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:18.397176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf6de8-aff9-473e-a41f-4f8bde47a0e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:18.465975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e2bf6de8-aff9-473e-a41f-4f8bde47a0e6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:18.467189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e2bf6de8-aff9-473e-a41f-4f8bde47a0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:18.480086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:18.566942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 711/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:18.864891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adbae41-8796-4770-8da1-a462d28213d2 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:18.866637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3adbae41-8796-4770-8da1-a462d28213d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:18.981659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adbae41-8796-4770-8da1-a462d28213d2 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:18.995485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:19.078300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f9406-8c25-49b4-a5fd-7f7188ac6e33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:19.078920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f9406-8c25-49b4-a5fd-7f7188ac6e33 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:19.079188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f9406-8c25-49b4-a5fd-7f7188ac6e33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:19.079581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f9406-8c25-49b4-a5fd-7f7188ac6e33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:19.081292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-961f9406-8c25-49b4-a5fd-7f7188ac6e33 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:19.210496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961f9406-8c25-49b4-a5fd-7f7188ac6e33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:19.211944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961f9406-8c25-49b4-a5fd-7f7188ac6e33 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:19.233818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 735/1446] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:21.538535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b2ad0-874a-4a52-9e73-c22d84a569e1 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:21.539410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b2ad0-874a-4a52-9e73-c22d84a569e1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:21.539410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b2ad0-874a-4a52-9e73-c22d84a569e1 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:21.540436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 712/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:21.546244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f55a62-38ac-4fff-a6d0-82540bed657b None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:21.546781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f55a62-38ac-4fff-a6d0-82540bed657b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:21.546781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f55a62-38ac-4fff-a6d0-82540bed657b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:21.599688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:21.620592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-30f55a62-38ac-4fff-a6d0-82540bed657b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:21.622569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-30f55a62-38ac-4fff-a6d0-82540bed657b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:21.733362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 736/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:22.051316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b2ad0-874a-4a52-9e73-c22d84a569e1 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:22.052868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-844b2ad0-874a-4a52-9e73-c22d84a569e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:22.172394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b2ad0-874a-4a52-9e73-c22d84a569e1 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:22.226256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:22.277433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f462fbb-5b4b-4cad-a0dc-92532b5ca7c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:22.278174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f462fbb-5b4b-4cad-a0dc-92532b5ca7c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:22.278631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f462fbb-5b4b-4cad-a0dc-92532b5ca7c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:22.278960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f462fbb-5b4b-4cad-a0dc-92532b5ca7c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:22.281192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9f462fbb-5b4b-4cad-a0dc-92532b5ca7c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:22.435778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f462fbb-5b4b-4cad-a0dc-92532b5ca7c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:22.437956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f462fbb-5b4b-4cad-a0dc-92532b5ca7c6 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:22.463684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 713/1449] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:24.570065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8078f43-0729-4745-8133-c8258fdc8d5b None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:24.570942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8078f43-0729-4745-8133-c8258fdc8d5b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:24.571079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8078f43-0729-4745-8133-c8258fdc8d5b None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:24.573159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 737/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:24.579149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73725cab-d567-49a1-b878-92bd5bf2e5fb None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:24.579648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73725cab-d567-49a1-b878-92bd5bf2e5fb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:24.579751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73725cab-d567-49a1-b878-92bd5bf2e5fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:24.652076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-73725cab-d567-49a1-b878-92bd5bf2e5fb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:24.653558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-73725cab-d567-49a1-b878-92bd5bf2e5fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:24.678743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:24.783441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 714/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:25.075106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8078f43-0729-4745-8133-c8258fdc8d5b None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:25.076842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b8078f43-0729-4745-8133-c8258fdc8d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:25.207597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8078f43-0729-4745-8133-c8258fdc8d5b None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:25.232055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:25.292107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5621643d-c12e-4718-b2e8-fc69376b1f63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:25.292713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5621643d-c12e-4718-b2e8-fc69376b1f63 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:25.292979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5621643d-c12e-4718-b2e8-fc69376b1f63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:25.293278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5621643d-c12e-4718-b2e8-fc69376b1f63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:25.295041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5621643d-c12e-4718-b2e8-fc69376b1f63 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:25.419131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5621643d-c12e-4718-b2e8-fc69376b1f63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:25.420952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5621643d-c12e-4718-b2e8-fc69376b1f63 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:25.443558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 738/1452] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:27.577359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf1d06-ddae-4d20-831e-d5edd07da7cf None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:27.578222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf1d06-ddae-4d20-831e-d5edd07da7cf None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:27.578222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf1d06-ddae-4d20-831e-d5edd07da7cf None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:27.579746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 715/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:27.585575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d71fc-e078-4014-802e-8723a7997ec4 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:27.586115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d71fc-e078-4014-802e-8723a7997ec4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:27.586115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747d71fc-e078-4014-802e-8723a7997ec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:27.660863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:27.663935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-747d71fc-e078-4014-802e-8723a7997ec4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:27.666076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-747d71fc-e078-4014-802e-8723a7997ec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:27.791388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 739/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:16:28.096998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf1d06-ddae-4d20-831e-d5edd07da7cf None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:28.098625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-cdaf1d06-ddae-4d20-831e-d5edd07da7cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:28.224831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf1d06-ddae-4d20-831e-d5edd07da7cf None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:28.282320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:28.319784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f41e351-d990-4d26-874f-8669814df891 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:28.320397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f41e351-d990-4d26-874f-8669814df891 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:28.320717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f41e351-d990-4d26-874f-8669814df891 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:28.320982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f41e351-d990-4d26-874f-8669814df891 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:28.322962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6f41e351-d990-4d26-874f-8669814df891 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:28.467349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f41e351-d990-4d26-874f-8669814df891 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:28.469295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f41e351-d990-4d26-874f-8669814df891 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:28.493915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 716/1455] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:30.628692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fc8dc7-5357-4466-822a-35121d0a6c54 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:30.629531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fc8dc7-5357-4466-822a-35121d0a6c54 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:30.629531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fc8dc7-5357-4466-822a-35121d0a6c54 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:30.630685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 740/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:30.637310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59439c7f-efb2-478c-b516-d810b601c116 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:30.637917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59439c7f-efb2-478c-b516-d810b601c116 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:30.637917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59439c7f-efb2-478c-b516-d810b601c116 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:30.739195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-59439c7f-efb2-478c-b516-d810b601c116 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:30.740499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-59439c7f-efb2-478c-b516-d810b601c116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:30.774849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:30.855894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 717/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:31.164765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fc8dc7-5357-4466-822a-35121d0a6c54 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:31.166451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-20fc8dc7-5357-4466-822a-35121d0a6c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:31.308683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fc8dc7-5357-4466-822a-35121d0a6c54 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:31.346368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:31.407903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d61e898-40da-4f62-b6a4-2b2cbd967ff5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:31.408401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d61e898-40da-4f62-b6a4-2b2cbd967ff5 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:31.408674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d61e898-40da-4f62-b6a4-2b2cbd967ff5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:31.408936 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d61e898-40da-4f62-b6a4-2b2cbd967ff5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:31.410613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0d61e898-40da-4f62-b6a4-2b2cbd967ff5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:31.545529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d61e898-40da-4f62-b6a4-2b2cbd967ff5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:31.548027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d61e898-40da-4f62-b6a4-2b2cbd967ff5 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:31.570533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 741/1458] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:16:33.604904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74761325-fc8d-41eb-8c82-7ffeae09d2ba None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:33.605579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74761325-fc8d-41eb-8c82-7ffeae09d2ba None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:33.605579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74761325-fc8d-41eb-8c82-7ffeae09d2ba None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:33.607617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 718/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:33.615573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44cdbe-5b00-42fc-8ab2-935bdd93a6c2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:33.616242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44cdbe-5b00-42fc-8ab2-935bdd93a6c2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:33.616242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44cdbe-5b00-42fc-8ab2-935bdd93a6c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:33.697581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5b44cdbe-5b00-42fc-8ab2-935bdd93a6c2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:33.698887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5b44cdbe-5b00-42fc-8ab2-935bdd93a6c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:33.708774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:33.824254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 742/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:34.118396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74761325-fc8d-41eb-8c82-7ffeae09d2ba None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:34.119960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-74761325-fc8d-41eb-8c82-7ffeae09d2ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:34.252657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74761325-fc8d-41eb-8c82-7ffeae09d2ba None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:34.328237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:34.351445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bde4d62-0f5e-4f5c-9820-507492b6b5f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:34.352056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde4d62-0f5e-4f5c-9820-507492b6b5f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:34.352511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde4d62-0f5e-4f5c-9820-507492b6b5f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:34.352783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde4d62-0f5e-4f5c-9820-507492b6b5f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:34.354512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7bde4d62-0f5e-4f5c-9820-507492b6b5f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:34.486358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bde4d62-0f5e-4f5c-9820-507492b6b5f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:34.487813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bde4d62-0f5e-4f5c-9820-507492b6b5f1 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:34.511985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 719/1461] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:36.573145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f81c00-3827-43a3-86c5-fc6095c7876a None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:36.573743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f81c00-3827-43a3-86c5-fc6095c7876a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:36.573743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f81c00-3827-43a3-86c5-fc6095c7876a None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:36.575098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 743/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:36.582068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23aabff-f538-49a8-a2d2-dab35b537adc None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:36.582833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23aabff-f538-49a8-a2d2-dab35b537adc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:36.582833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23aabff-f538-49a8-a2d2-dab35b537adc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:36.666499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c23aabff-f538-49a8-a2d2-dab35b537adc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:36.667750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c23aabff-f538-49a8-a2d2-dab35b537adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:36.716128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:36.806537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 720/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:37.105986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f81c00-3827-43a3-86c5-fc6095c7876a None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:37.108254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-12f81c00-3827-43a3-86c5-fc6095c7876a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:37.240745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f81c00-3827-43a3-86c5-fc6095c7876a None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:37.291267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:37.338136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b841e89-fa59-43a4-b749-1b1de2160e3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:37.338751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b841e89-fa59-43a4-b749-1b1de2160e3b service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:37.339057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b841e89-fa59-43a4-b749-1b1de2160e3b service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:37.339300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b841e89-fa59-43a4-b749-1b1de2160e3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:37.343702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8b841e89-fa59-43a4-b749-1b1de2160e3b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:37.471622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b841e89-fa59-43a4-b749-1b1de2160e3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:37.472977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b841e89-fa59-43a4-b749-1b1de2160e3b service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:37.492902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 744/1464] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:39.576026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce7fb0a-8cef-4586-8750-3994e975e96e None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:39.576805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce7fb0a-8cef-4586-8750-3994e975e96e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:39.576805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce7fb0a-8cef-4586-8750-3994e975e96e None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:39.579136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 721/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:39.587244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb722c-0f52-4641-86c1-9e74f3b66a0a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:39.587781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb722c-0f52-4641-86c1-9e74f3b66a0a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:39.587781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb722c-0f52-4641-86c1-9e74f3b66a0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:39.666233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-14fb722c-0f52-4641-86c1-9e74f3b66a0a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:39.667480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-14fb722c-0f52-4641-86c1-9e74f3b66a0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:39.688258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:39.782631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 745/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:40.071552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce7fb0a-8cef-4586-8750-3994e975e96e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:40.074111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2ce7fb0a-8cef-4586-8750-3994e975e96e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:40.213274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce7fb0a-8cef-4586-8750-3994e975e96e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:40.290428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:40.297907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8745e36b-b3c2-4413-9457-8a3d95708fb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:40.298514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8745e36b-b3c2-4413-9457-8a3d95708fb3 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:40.298783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8745e36b-b3c2-4413-9457-8a3d95708fb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:40.299055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8745e36b-b3c2-4413-9457-8a3d95708fb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:40.300690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8745e36b-b3c2-4413-9457-8a3d95708fb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:40.432616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8745e36b-b3c2-4413-9457-8a3d95708fb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:40.434180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8745e36b-b3c2-4413-9457-8a3d95708fb3 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:40.458558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 722/1467] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:16:42.546694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3492ef2f-2cd8-4ba5-821a-aa9a86544399 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:42.547234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3492ef2f-2cd8-4ba5-821a-aa9a86544399 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:42.547234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3492ef2f-2cd8-4ba5-821a-aa9a86544399 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:42.548875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 746/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:16:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:16:42.555699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd437d7-452b-4eb6-a378-896ca3eecea0 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:42.556265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd437d7-452b-4eb6-a378-896ca3eecea0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:42.556265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd437d7-452b-4eb6-a378-896ca3eecea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:42.633340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2cd437d7-452b-4eb6-a378-896ca3eecea0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:16:42.634577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2cd437d7-452b-4eb6-a378-896ca3eecea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:42.692905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:42.748870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 723/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:16:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:16:43.045529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3492ef2f-2cd8-4ba5-821a-aa9a86544399 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:16:43.047050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3492ef2f-2cd8-4ba5-821a-aa9a86544399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:43.164882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3492ef2f-2cd8-4ba5-821a-aa9a86544399 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:16:43.224226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:43.257597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd663d63-7148-4a65-a288-48db67fa8373 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:16:43.258241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd663d63-7148-4a65-a288-48db67fa8373 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:16:43.258575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd663d63-7148-4a65-a288-48db67fa8373 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:16:43.258890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd663d63-7148-4a65-a288-48db67fa8373 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:16:43.260586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fd663d63-7148-4a65-a288-48db67fa8373 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:16:43.406746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd663d63-7148-4a65-a288-48db67fa8373 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:16:43.406746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd663d63-7148-4a65-a288-48db67fa8373 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:16:43.428069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 747/1470] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:16:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:16:52.746446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=22 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:16:53.425212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=66 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:04.743001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a6f1c-d511-4695-87f6-3f0749f269ab None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:04.743568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a6f1c-d511-4695-87f6-3f0749f269ab None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:04.743568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a6f1c-d511-4695-87f6-3f0749f269ab None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:04.744829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 724/1471] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:17:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:17:04.754060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e0e20-c65f-4769-9751-09f969f265cd None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:04.754622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e0e20-c65f-4769-9751-09f969f265cd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:04.754965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e0e20-c65f-4769-9751-09f969f265cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:04.834829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-291e0e20-c65f-4769-9751-09f969f265cd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:04.836031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-291e0e20-c65f-4769-9751-09f969f265cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:04.960656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 748/1472] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:17:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:17:05.262862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d1c316-5b5a-4d60-8523-c3deb42da0af None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:05.263139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d1c316-5b5a-4d60-8523-c3deb42da0af None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:05.263410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d1c316-5b5a-4d60-8523-c3deb42da0af None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:05.265153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 725/1473] 10.222.0.52 () {54 vars in 967 bytes} [Sat Aug 27 02:17:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:17:05.272448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494f930-7b41-4671-99ed-9c48df42bb71 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:05.272733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494f930-7b41-4671-99ed-9c48df42bb71 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:05.273040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494f930-7b41-4671-99ed-9c48df42bb71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:05.380925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-8494f930-7b41-4671-99ed-9c48df42bb71 None None] MFA Rules not processed for user `ab71b242a16641429c29dc8aa67867e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:05.382214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8494f930-7b41-4671-99ed-9c48df42bb71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:05.407296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:05.502197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 749/1474] 10.222.0.52 () {58 vars in 1064 bytes} [Sat Aug 27 02:17:05 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:17:05.510019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d1c316-5b5a-4d60-8523-c3deb42da0af None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:17:05.511543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c0d1c316-5b5a-4d60-8523-c3deb42da0af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:05.650765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d1c316-5b5a-4d60-8523-c3deb42da0af None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:17:05.758533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd6ad36-5ae3-4814-bc6b-1dab67d385ad service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:17:05.759228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd6ad36-5ae3-4814-bc6b-1dab67d385ad service glance] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:05.759349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd6ad36-5ae3-4814-bc6b-1dab67d385ad service glance] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:05.759552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd6ad36-5ae3-4814-bc6b-1dab67d385ad service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:05.761174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4bd6ad36-5ae3-4814-bc6b-1dab67d385ad service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:05.840635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:05.921721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd6ad36-5ae3-4814-bc6b-1dab67d385ad service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:17:05.923192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd6ad36-5ae3-4814-bc6b-1dab67d385ad service glance] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:17:05.943910 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 726/1475] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:17:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:17:08.079892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c4e633-1fe3-4a7c-b76d-76575736a223 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:08.080761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c4e633-1fe3-4a7c-b76d-76575736a223 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:08.080761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c4e633-1fe3-4a7c-b76d-76575736a223 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:08.085328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 750/1476] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:17:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:17:08.088089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ebc09a-bf02-486c-addb-aaac7729c6a2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:08.088577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ebc09a-bf02-486c-addb-aaac7729c6a2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:08.088577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ebc09a-bf02-486c-addb-aaac7729c6a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:08.181153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b5ebc09a-bf02-486c-addb-aaac7729c6a2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:08.182729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b5ebc09a-bf02-486c-addb-aaac7729c6a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:08.296453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 727/1477] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:17:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:17:08.647823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c4e633-1fe3-4a7c-b76d-76575736a223 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:17:08.649436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-44c4e633-1fe3-4a7c-b76d-76575736a223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:08.803475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c4e633-1fe3-4a7c-b76d-76575736a223 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:17:08.868450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:08.943295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3571eba8-44b6-4cbb-97b4-26f0127bb22f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:17:08.943915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3571eba8-44b6-4cbb-97b4-26f0127bb22f admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:08.944188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3571eba8-44b6-4cbb-97b4-26f0127bb22f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:08.944452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3571eba8-44b6-4cbb-97b4-26f0127bb22f admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:08.945109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3571eba8-44b6-4cbb-97b4-26f0127bb22f admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:17:08.946980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-3571eba8-44b6-4cbb-97b4-26f0127bb22f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:17:08.946980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:17:08.946980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:17:08.948416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3571eba8-44b6-4cbb-97b4-26f0127bb22f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:17:08.972882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 751/1478] 10.222.0.52 () {56 vars in 1108 bytes} [Sat Aug 27 02:17:08 2022] GET /identity/v3/projects => generated 2553 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:17:10.988860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a61a95-5639-41f6-87ca-d150e43b2e0f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:10.989575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a61a95-5639-41f6-87ca-d150e43b2e0f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:10.989575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a61a95-5639-41f6-87ca-d150e43b2e0f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:10.990847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 728/1479] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:17:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:17:10.999571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5830fa-b752-4710-a297-1aa903fae222 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:11.000288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5830fa-b752-4710-a297-1aa903fae222 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:11.000288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5830fa-b752-4710-a297-1aa903fae222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:11.079068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-af5830fa-b752-4710-a297-1aa903fae222 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:11.080303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-af5830fa-b752-4710-a297-1aa903fae222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:11.198563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 752/1480] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:17:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:17:11.507814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a61a95-5639-41f6-87ca-d150e43b2e0f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:17:11.509699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b2a61a95-5639-41f6-87ca-d150e43b2e0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:11.522331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:11.714211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a61a95-5639-41f6-87ca-d150e43b2e0f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:17:11.828658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:11.868730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f8d789-4770-412b-a0ed-7c49cd01d1aa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:17:11.869415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d789-4770-412b-a0ed-7c49cd01d1aa service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:11.869686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d789-4770-412b-a0ed-7c49cd01d1aa service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:11.869951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d789-4770-412b-a0ed-7c49cd01d1aa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:11.871589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-55f8d789-4770-412b-a0ed-7c49cd01d1aa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:11.981595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f8d789-4770-412b-a0ed-7c49cd01d1aa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:17:11.983368 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f8d789-4770-412b-a0ed-7c49cd01d1aa service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:17:12.013622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 729/1481] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:17:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:17:14.300610 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e189a-8224-4cf4-8870-65131bd6e4f5 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:14.301227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e189a-8224-4cf4-8870-65131bd6e4f5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:14.301699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e189a-8224-4cf4-8870-65131bd6e4f5 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:14.304265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 753/1482] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:17:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:17:14.311254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506c45e4-bed4-4d01-b622-e5b44ff364c9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:14.311935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506c45e4-bed4-4d01-b622-e5b44ff364c9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:14.312036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506c45e4-bed4-4d01-b622-e5b44ff364c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:14.408822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-506c45e4-bed4-4d01-b622-e5b44ff364c9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:14.410495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-506c45e4-bed4-4d01-b622-e5b44ff364c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:14.482641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:14.557139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 730/1483] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:17:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:17:14.871852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e189a-8224-4cf4-8870-65131bd6e4f5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:17:14.874086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a63e189a-8224-4cf4-8870-65131bd6e4f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:14.910977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:15.022448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e189a-8224-4cf4-8870-65131bd6e4f5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:17:15.112252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baf3e53-b42f-43f3-bccd-59db91ba9217 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:17:15.113400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baf3e53-b42f-43f3-bccd-59db91ba9217 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:15.113879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baf3e53-b42f-43f3-bccd-59db91ba9217 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:15.114345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baf3e53-b42f-43f3-bccd-59db91ba9217 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:15.117441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7baf3e53-b42f-43f3-bccd-59db91ba9217 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:15.216341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:15.255539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baf3e53-b42f-43f3-bccd-59db91ba9217 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:17:15.257175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baf3e53-b42f-43f3-bccd-59db91ba9217 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:17:15.282952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 754/1484] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:17:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:17:17.311940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0581772-954a-40b3-a8ba-cbe627ec4dd5 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:17.312543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0581772-954a-40b3-a8ba-cbe627ec4dd5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:17.312638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0581772-954a-40b3-a8ba-cbe627ec4dd5 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:17.314135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 731/1485] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:17:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:17:17.320051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf24838e-2daa-4074-accc-5879bc790ab1 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:17.320640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf24838e-2daa-4074-accc-5879bc790ab1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:17.320640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf24838e-2daa-4074-accc-5879bc790ab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:17.403644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-bf24838e-2daa-4074-accc-5879bc790ab1 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:17.404849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bf24838e-2daa-4074-accc-5879bc790ab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:17.544852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 755/1486] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:17:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:17:17.837758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0581772-954a-40b3-a8ba-cbe627ec4dd5 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:17:17.839682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c0581772-954a-40b3-a8ba-cbe627ec4dd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:17.962512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0581772-954a-40b3-a8ba-cbe627ec4dd5 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:17:18.050321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:18.052502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecbf07a-5182-43f4-8245-2d52495bea08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:17:18.053175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbf07a-5182-43f4-8245-2d52495bea08 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:18.053684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbf07a-5182-43f4-8245-2d52495bea08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:18.053999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbf07a-5182-43f4-8245-2d52495bea08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:18.055722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2ecbf07a-5182-43f4-8245-2d52495bea08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:18.178938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecbf07a-5182-43f4-8245-2d52495bea08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:17:18.180379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecbf07a-5182-43f4-8245-2d52495bea08 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:17:18.201461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 732/1487] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:17:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:17:20.418035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a80b51-b832-4f81-85ee-2171d94c4cab None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:20.418686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a80b51-b832-4f81-85ee-2171d94c4cab None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:20.418686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a80b51-b832-4f81-85ee-2171d94c4cab None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:20.419914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 756/1488] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:17:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:17:20.426219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62de779-3a35-400d-9c0f-42aa653488c3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:20.426730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62de779-3a35-400d-9c0f-42aa653488c3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:20.426730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62de779-3a35-400d-9c0f-42aa653488c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:20.502205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b62de779-3a35-400d-9c0f-42aa653488c3 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:20.503437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b62de779-3a35-400d-9c0f-42aa653488c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:20.572476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:20.633752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 733/1489] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:17:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:17:20.925913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a80b51-b832-4f81-85ee-2171d94c4cab None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:17:20.927614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d1a80b51-b832-4f81-85ee-2171d94c4cab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:20.972585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:21.049952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a80b51-b832-4f81-85ee-2171d94c4cab None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:17:21.131212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbf8ef0-9aa1-4613-b5a1-c288b38564f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:17:21.131799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbf8ef0-9aa1-4613-b5a1-c288b38564f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:21.132069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbf8ef0-9aa1-4613-b5a1-c288b38564f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:21.132330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbf8ef0-9aa1-4613-b5a1-c288b38564f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:21.134221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-acbf8ef0-9aa1-4613-b5a1-c288b38564f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:21.231516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:21.264234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbf8ef0-9aa1-4613-b5a1-c288b38564f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:17:21.265912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbf8ef0-9aa1-4613-b5a1-c288b38564f2 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:17:21.285348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 757/1490] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:17:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:17:23.569902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa3e74-74a6-41f1-985c-7291dff2d2fe None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:23.570553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa3e74-74a6-41f1-985c-7291dff2d2fe None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:23.570553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa3e74-74a6-41f1-985c-7291dff2d2fe None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:23.571935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 734/1491] 10.222.0.52 () {54 vars in 927 bytes} [Sat Aug 27 02:17:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:17:23.578625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bafcc47-5115-47c6-8112-1f750367ca92 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:23.579251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bafcc47-5115-47c6-8112-1f750367ca92 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:23.579251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bafcc47-5115-47c6-8112-1f750367ca92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:23.662338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-7bafcc47-5115-47c6-8112-1f750367ca92 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:17:23.663631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7bafcc47-5115-47c6-8112-1f750367ca92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:23.784967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 758/1492] 10.222.0.52 () {58 vars in 1024 bytes} [Sat Aug 27 02:17:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:17:24.081696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaa3e74-74a6-41f1-985c-7291dff2d2fe None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:17:24.083349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0aaa3e74-74a6-41f1-985c-7291dff2d2fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:24.222691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaa3e74-74a6-41f1-985c-7291dff2d2fe None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:17:24.308434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d53be6-eb0b-4f36-8ab1-f4883cee1d6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:17:24.309057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d53be6-eb0b-4f36-8ab1-f4883cee1d6b service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:17:24.309351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d53be6-eb0b-4f36-8ab1-f4883cee1d6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:17:24.309616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d53be6-eb0b-4f36-8ab1-f4883cee1d6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:17:24.311297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a7d53be6-eb0b-4f36-8ab1-f4883cee1d6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:17:24.327351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:24.491858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d53be6-eb0b-4f36-8ab1-f4883cee1d6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:17:24.493533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d53be6-eb0b-4f36-8ab1-f4883cee1d6b service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:17:24.521689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 735/1493] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:17:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:17:33.782998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=82 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:17:34.513553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=50 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:45.870382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:45.871069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:45.871069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:45.970942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:18:45.972109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:46.080756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 759/1494] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:18:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:18:46.089484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1343886b-8ff6-4e55-880a-17d657ab1751 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:46.090053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1343886b-8ff6-4e55-880a-17d657ab1751 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:46.090053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1343886b-8ff6-4e55-880a-17d657ab1751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:46.200483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1343886b-8ff6-4e55-880a-17d657ab1751 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:18:46.201982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1343886b-8ff6-4e55-880a-17d657ab1751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:46.323138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 736/1495] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:18:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:18:46.328128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:46.329402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:46.342337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b8076-3f6b-4d20-981f-809cad88445b None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:46.425491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:46.470763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:46.471388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:46.471662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:46.471934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:46.472663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:18:46.472971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:46.474056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:46.474056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:18:46.474056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:18:46.474963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:46.488302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 760/1496] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:18:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:18:46.497263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1343886b-8ff6-4e55-880a-17d657ab1751 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:46.507047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1343886b-8ff6-4e55-880a-17d657ab1751 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:46.600522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:46.646561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:46.647192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:46.647469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:46.647749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:46.648418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:46.650271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING py.warnings [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:46.650271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: warnings.warn(msg) Aug 27 02:18:46.650271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: Aug 27 02:18:46.651688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:46.722217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 737/1497] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:18:46 2022] POST /identity/v3/projects => generated 377 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:18:46.729088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:46.731850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:46.772584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4476c3a5-ce7a-4807-ae55-941ee43a04e5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:46.934181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:46.934871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:46.934976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:46.935221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:46.935872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:46.936918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:46.936918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:18:46.936918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:18:46.938183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:46.989915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 761/1498] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:18:46 2022] POST /identity/v3/users => generated 289 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:18:46.994400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:46.996021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:47.072667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672a61c6-ee16-4409-88bb-1107e581629e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:47.172445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:47.186991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:47.187537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:47.187809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:47.188071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:47.188998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:47.190086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING py.warnings [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:47.190086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: warnings.warn(msg) Aug 27 02:18:47.190086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: Aug 27 02:18:47.190990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:47.205896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 738/1499] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:18:47.210718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:47.212386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:47.225603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ef0376-c8d9-4ffa-8f96-ffce201eee2b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:47.256763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:47.345289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8320ff5c-ab9b-46e4-ac85-c3f112d0a285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:47.348317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8320ff5c-ab9b-46e4-ac85-c3f112d0a285 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:47.348317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8320ff5c-ab9b-46e4-ac85-c3f112d0a285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:47.348317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8320ff5c-ab9b-46e4-ac85-c3f112d0a285 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:47.348317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8320ff5c-ab9b-46e4-ac85-c3f112d0a285 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:47.348317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-8320ff5c-ab9b-46e4-ac85-c3f112d0a285 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:47.348317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:18:47.348317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:18:47.350011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8320ff5c-ab9b-46e4-ac85-c3f112d0a285 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:47.363924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 762/1500] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:18:47.368385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:47.369950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:47.405803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a811a-6591-475b-ad42-19aff4fd14ce admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:47.531623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eac56d-14e9-48b6-95e9-8fc2eeb7c843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:47.532502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eac56d-14e9-48b6-95e9-8fc2eeb7c843 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:47.532925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eac56d-14e9-48b6-95e9-8fc2eeb7c843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:47.533403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eac56d-14e9-48b6-95e9-8fc2eeb7c843 admin admin] PATH_INFO: `/v3/projects/e2891c77c6864ecb8600af5d1902acec/users/fbfc4f0db8344642aeefe13734c14e10/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:47.571239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eac56d-14e9-48b6-95e9-8fc2eeb7c843 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2891c77c6864ecb8600af5d1902acec, user_id=fbfc4f0db8344642aeefe13734c14e10, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:47.576277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING py.warnings [None req-e2eac56d-14e9-48b6-95e9-8fc2eeb7c843 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:47.576277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: warnings.warn(msg) Aug 27 02:18:47.576277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: Aug 27 02:18:47.576277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eac56d-14e9-48b6-95e9-8fc2eeb7c843 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:47.608402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 739/1501] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:18:47 2022] PUT /identity/v3/projects/e2891c77c6864ecb8600af5d1902acec/users/fbfc4f0db8344642aeefe13734c14e10/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:18:47.622372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621e6595-b914-4f70-86cb-d8eacd5c54a0 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:47.622688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621e6595-b914-4f70-86cb-d8eacd5c54a0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:47.622978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621e6595-b914-4f70-86cb-d8eacd5c54a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:47.690835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-621e6595-b914-4f70-86cb-d8eacd5c54a0 None None] MFA Rules not processed for user `fbfc4f0db8344642aeefe13734c14e10`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:18:47.692446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-621e6595-b914-4f70-86cb-d8eacd5c54a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:47.835253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 763/1502] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:18:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:18:47.850683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7c66bb-ba48-4984-bcb0-fa5dc80dc285 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:47.850989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7c66bb-ba48-4984-bcb0-fa5dc80dc285 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:47.851297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7c66bb-ba48-4984-bcb0-fa5dc80dc285 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:47.853441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 740/1503] 10.222.0.52 () {54 vars in 964 bytes} [Sat Aug 27 02:18:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:18:47.860920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f39a5ef-3fb8-4ba7-89ae-d2026e85a629 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:47.861245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f39a5ef-3fb8-4ba7-89ae-d2026e85a629 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:47.861507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f39a5ef-3fb8-4ba7-89ae-d2026e85a629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:47.905473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:47.959330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6f39a5ef-3fb8-4ba7-89ae-d2026e85a629 None None] MFA Rules not processed for user `598bc689672741c0ac600e42cf45e39d`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:18:47.960569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6f39a5ef-3fb8-4ba7-89ae-d2026e85a629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:48.053720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 764/1504] 10.222.0.52 () {58 vars in 1061 bytes} [Sat Aug 27 02:18:47 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:18:48.061498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7c66bb-ba48-4984-bcb0-fa5dc80dc285 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:48.063020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2e7c66bb-ba48-4984-bcb0-fa5dc80dc285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:48.126119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:48.193831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7c66bb-ba48-4984-bcb0-fa5dc80dc285 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:48.284432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:48.285141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:48.285645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:48.285914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:48.287635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:48.399266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:48.400627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:48.404425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:48.428907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 741/1505] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:18:48 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:18:48.483514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e54b14-b11c-46f7-88ba-bcdeaab08bc6 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:48.483814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e54b14-b11c-46f7-88ba-bcdeaab08bc6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:48.484078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e54b14-b11c-46f7-88ba-bcdeaab08bc6 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:48.485394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 765/1506] 10.222.0.52 () {52 vars in 830 bytes} [Sat Aug 27 02:18:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:18:48.490093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:48.491670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:48.609312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d86c7-e756-421a-a7fb-6ffbf271623b service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:48.720317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:48.721270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:48.721817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:48.722215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] PATH_INFO: `/v3/users/fbfc4f0db8344642aeefe13734c14e10` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:48.738033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbfc4f0db8344642aeefe13734c14e10)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:48.739365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING py.warnings [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:48.739365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: warnings.warn(msg) Aug 27 02:18:48.739365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: Aug 27 02:18:48.740587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:48.751665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:48.822778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbfc4f0db8344642aeefe13734c14e10'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:48.841466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbfc4f0db8344642aeefe13734c14e10'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:48.854090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbfc4f0db8344642aeefe13734c14e10'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:48.854780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbfc4f0db8344642aeefe13734c14e10'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:48.855188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbfc4f0db8344642aeefe13734c14e10'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:48.891682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 742/1507] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:18:48 2022] DELETE /identity/v3/users/fbfc4f0db8344642aeefe13734c14e10 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:18:48.897606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:48.897951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:48.898405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:48.962938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:18:48.964469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:49.058618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:49.075247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 766/1508] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:18:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:18:49.088915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:49.089905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:49.208872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976097e0-2e99-4db8-9e6d-38fe4024fe3d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:49.284717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:18:49.312622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff354b9e-136c-4e29-bc55-372a21b9e6b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:49.313282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff354b9e-136c-4e29-bc55-372a21b9e6b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:49.313560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff354b9e-136c-4e29-bc55-372a21b9e6b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:49.313825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff354b9e-136c-4e29-bc55-372a21b9e6b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:49.315539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ff354b9e-136c-4e29-bc55-372a21b9e6b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:49.460190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff354b9e-136c-4e29-bc55-372a21b9e6b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:49.462026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff354b9e-136c-4e29-bc55-372a21b9e6b5 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:49.483826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 743/1509] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:18:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:18:49.857928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:18:49.859706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:18:49.872748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df35e61b-d543-47ab-808a-6e1d5da9d8bc None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:18:49.988134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:18:49.989040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:18:49.989472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:18:49.989748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] PATH_INFO: `/v3/projects/e2891c77c6864ecb8600af5d1902acec` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:18:49.999900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2891c77c6864ecb8600af5d1902acec)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:18:50.001303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:18:50.001303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:18:50.001303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:18:50.002909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:18:50.069041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2891c77c6864ecb8600af5d1902acec', 'request_id': 'req-d0b44953-542f-4228-87ee-379f63b9a0a3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:50.084519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2891c77c6864ecb8600af5d1902acec', 'request_id': 'req-d0b44953-542f-4228-87ee-379f63b9a0a3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:50.091251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2891c77c6864ecb8600af5d1902acec', 'request_id': 'req-d0b44953-542f-4228-87ee-379f63b9a0a3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:50.241928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] The token cache is being invalidate because project e2891c77c6864ecb8600af5d1902acec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:18:50.242162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:50.242637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d0b44953-542f-4228-87ee-379f63b9a0a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:18:50.243961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 767/1510] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:18:49 2022] DELETE /identity/v3/projects/e2891c77c6864ecb8600af5d1902acec => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:18:59.481278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=62 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:19:00.239921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:20:21.259072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada1198-d08e-4144-b5cc-cca24f65a7f9 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:21.259755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada1198-d08e-4144-b5cc-cca24f65a7f9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:21.259755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada1198-d08e-4144-b5cc-cca24f65a7f9 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:21.260976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 744/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:20:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:21.266472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6682fa28-4d76-4a29-8a09-a80069e9afc5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:21.267031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6682fa28-4d76-4a29-8a09-a80069e9afc5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:21.267031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6682fa28-4d76-4a29-8a09-a80069e9afc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:21.374206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6682fa28-4d76-4a29-8a09-a80069e9afc5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:21.375096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6682fa28-4d76-4a29-8a09-a80069e9afc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:21.507417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 768/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:20:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:21.662615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ada1198-d08e-4144-b5cc-cca24f65a7f9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:21.665724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6ada1198-d08e-4144-b5cc-cca24f65a7f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:21.820868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ada1198-d08e-4144-b5cc-cca24f65a7f9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:21.828385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79351e9-760d-47ef-85a5-4c2411edf588 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:21.829344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79351e9-760d-47ef-85a5-4c2411edf588 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:21.829617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79351e9-760d-47ef-85a5-4c2411edf588 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:21.829901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79351e9-760d-47ef-85a5-4c2411edf588 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:21.839017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79351e9-760d-47ef-85a5-4c2411edf588 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:21.840611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79351e9-760d-47ef-85a5-4c2411edf588 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:21.850865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 745/1513] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:22.720422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012214c1-838c-4741-878e-40a353853f51 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:22.721240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012214c1-838c-4741-878e-40a353853f51 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:22.721433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012214c1-838c-4741-878e-40a353853f51 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:22.723201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 769/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:20:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:22.729359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9b3ad-0c30-4b54-8509-320ee52a121e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:22.730186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9b3ad-0c30-4b54-8509-320ee52a121e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:22.730365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9b3ad-0c30-4b54-8509-320ee52a121e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:22.752659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8fa9b3ad-0c30-4b54-8509-320ee52a121e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:22.753952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8fa9b3ad-0c30-4b54-8509-320ee52a121e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:22.762818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 746/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:20:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:22.928135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012214c1-838c-4741-878e-40a353853f51 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:22.948229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012214c1-838c-4741-878e-40a353853f51 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:22.957192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0cf26a-529c-4d18-8326-87a74a242533 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:22.958033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0cf26a-529c-4d18-8326-87a74a242533 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:22.958549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0cf26a-529c-4d18-8326-87a74a242533 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:22.958984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0cf26a-529c-4d18-8326-87a74a242533 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:22.970927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0cf26a-529c-4d18-8326-87a74a242533 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:22.972540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0cf26a-529c-4d18-8326-87a74a242533 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:22.983493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 770/1516] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:20:23.135554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b199c4-93d6-4ac1-85ab-5ed0efadad31 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:23.136025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b199c4-93d6-4ac1-85ab-5ed0efadad31 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:23.136461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b199c4-93d6-4ac1-85ab-5ed0efadad31 None None] PATH_INFO: `/v3` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:23.138692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 747/1517] 10.222.0.45 () {54 vars in 925 bytes} [Sat Aug 27 02:20:23 2022] GET /identity/v3 => generated 255 bytes in 4 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:20:23.145184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29de1913-0c96-4e87-a5bb-16d2bb02bd36 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:23.145577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29de1913-0c96-4e87-a5bb-16d2bb02bd36 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:23.145747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29de1913-0c96-4e87-a5bb-16d2bb02bd36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:23.214531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-29de1913-0c96-4e87-a5bb-16d2bb02bd36 None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:23.215774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-29de1913-0c96-4e87-a5bb-16d2bb02bd36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:23.240726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:20:23.310392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 771/1518] 10.222.0.45 () {58 vars in 1015 bytes} [Sat Aug 27 02:20:23 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:32.570759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d541fe5-044a-427a-987a-8b53a8a50fd9 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:32.571303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d541fe5-044a-427a-987a-8b53a8a50fd9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:32.571303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d541fe5-044a-427a-987a-8b53a8a50fd9 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:32.572697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 748/1519] 10.222.0.52 () {54 vars in 961 bytes} [Sat Aug 27 02:20:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:32.578694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9cf54-e64f-4ec1-be2d-f7762235fe2a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:32.579166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9cf54-e64f-4ec1-be2d-f7762235fe2a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:32.579272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9cf54-e64f-4ec1-be2d-f7762235fe2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:32.604470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-9be9cf54-e64f-4ec1-be2d-f7762235fe2a None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:32.606354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9be9cf54-e64f-4ec1-be2d-f7762235fe2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:32.616408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 772/1520] 10.222.0.52 () {58 vars in 1058 bytes} [Sat Aug 27 02:20:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:32.624473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d541fe5-044a-427a-987a-8b53a8a50fd9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:32.639398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d541fe5-044a-427a-987a-8b53a8a50fd9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:32.647132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dffecf-ba85-40b6-8050-38fe0ac762ab service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:32.648099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dffecf-ba85-40b6-8050-38fe0ac762ab service placement] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:32.648565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dffecf-ba85-40b6-8050-38fe0ac762ab service placement] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:32.648907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dffecf-ba85-40b6-8050-38fe0ac762ab service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:32.661464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dffecf-ba85-40b6-8050-38fe0ac762ab service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:32.663082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dffecf-ba85-40b6-8050-38fe0ac762ab service placement] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:32.674163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 749/1521] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:34.272740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd67ee19-8d96-4361-98ec-2df6d04d26fa None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:34.273469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd67ee19-8d96-4361-98ec-2df6d04d26fa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:34.273469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd67ee19-8d96-4361-98ec-2df6d04d26fa None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:34.274716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 773/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:20:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:34.280601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b6f5d4-4a80-4481-880d-4bafcd98e177 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:34.281175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b6f5d4-4a80-4481-880d-4bafcd98e177 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:34.281175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b6f5d4-4a80-4481-880d-4bafcd98e177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:34.304400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-63b6f5d4-4a80-4481-880d-4bafcd98e177 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:34.305674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-63b6f5d4-4a80-4481-880d-4bafcd98e177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:34.315218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 750/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:20:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:34.475336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd67ee19-8d96-4361-98ec-2df6d04d26fa None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:34.490833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd67ee19-8d96-4361-98ec-2df6d04d26fa None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:34.496484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c67430-e1cc-48bb-90a1-a674c0ba66e1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:34.497097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c67430-e1cc-48bb-90a1-a674c0ba66e1 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:34.497426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c67430-e1cc-48bb-90a1-a674c0ba66e1 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:34.497683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c67430-e1cc-48bb-90a1-a674c0ba66e1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:34.507039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c67430-e1cc-48bb-90a1-a674c0ba66e1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:34.508638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c67430-e1cc-48bb-90a1-a674c0ba66e1 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:34.520056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 774/1524] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:35.374593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024a001f-b4ae-40ed-83d3-20acd2167fd4 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:35.375222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024a001f-b4ae-40ed-83d3-20acd2167fd4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:35.375222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024a001f-b4ae-40ed-83d3-20acd2167fd4 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:35.376500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 751/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sat Aug 27 02:20:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:20:35.382191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54854c1-fc5a-4081-94ec-54e05ee89f2a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:35.382745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54854c1-fc5a-4081-94ec-54e05ee89f2a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:35.382745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54854c1-fc5a-4081-94ec-54e05ee89f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:35.407512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c54854c1-fc5a-4081-94ec-54e05ee89f2a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:35.408725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c54854c1-fc5a-4081-94ec-54e05ee89f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:35.419292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 775/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sat Aug 27 02:20:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:35.574474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024a001f-b4ae-40ed-83d3-20acd2167fd4 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:35.590459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024a001f-b4ae-40ed-83d3-20acd2167fd4 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:35.597446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb8556c-99d8-421d-8dd7-032305bee50f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:35.598053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb8556c-99d8-421d-8dd7-032305bee50f service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:35.598329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb8556c-99d8-421d-8dd7-032305bee50f service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:35.598595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb8556c-99d8-421d-8dd7-032305bee50f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:35.607988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb8556c-99d8-421d-8dd7-032305bee50f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:35.609621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb8556c-99d8-421d-8dd7-032305bee50f service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:35.619880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 752/1527] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:39.568258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a2e444-e202-456d-9b4a-6ff7001216ca None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:39.568904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a2e444-e202-456d-9b4a-6ff7001216ca None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:39.568904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a2e444-e202-456d-9b4a-6ff7001216ca None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:39.570147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 776/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:20:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:39.577282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1975b25-fbfa-4364-82b6-567967158c03 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:39.578153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1975b25-fbfa-4364-82b6-567967158c03 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:39.578153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1975b25-fbfa-4364-82b6-567967158c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:39.599980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c1975b25-fbfa-4364-82b6-567967158c03 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:39.601703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c1975b25-fbfa-4364-82b6-567967158c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:39.612940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 753/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:20:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:39.895889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a2e444-e202-456d-9b4a-6ff7001216ca None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:39.897713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-50a2e444-e202-456d-9b4a-6ff7001216ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:39.987822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a2e444-e202-456d-9b4a-6ff7001216ca None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:39.991906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37706d7d-4267-408f-b417-fa3fe17b5010 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:39.992582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37706d7d-4267-408f-b417-fa3fe17b5010 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:39.992917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37706d7d-4267-408f-b417-fa3fe17b5010 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:39.993263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37706d7d-4267-408f-b417-fa3fe17b5010 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:40.004123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37706d7d-4267-408f-b417-fa3fe17b5010 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:40.005600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37706d7d-4267-408f-b417-fa3fe17b5010 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:40.022399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 777/1530] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:41.978651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9e11a-8a9e-4485-b557-7c00daf3ab9f None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:41.979204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9e11a-8a9e-4485-b557-7c00daf3ab9f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:41.979204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9e11a-8a9e-4485-b557-7c00daf3ab9f None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:41.980809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 754/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:20:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:41.987148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8799eb-70de-4fa5-a6cd-bbb3c430cbd5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:41.987657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8799eb-70de-4fa5-a6cd-bbb3c430cbd5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:41.987657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8799eb-70de-4fa5-a6cd-bbb3c430cbd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:42.007372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-af8799eb-70de-4fa5-a6cd-bbb3c430cbd5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:42.008556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-af8799eb-70de-4fa5-a6cd-bbb3c430cbd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:42.017013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 778/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:20:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:42.307067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e9e11a-8a9e-4485-b557-7c00daf3ab9f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:42.319444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e9e11a-8a9e-4485-b557-7c00daf3ab9f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:42.323576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c155478b-1ccc-4370-8b24-c4a2d22dda00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:42.324217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c155478b-1ccc-4370-8b24-c4a2d22dda00 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:42.324523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c155478b-1ccc-4370-8b24-c4a2d22dda00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:42.324826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c155478b-1ccc-4370-8b24-c4a2d22dda00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:42.336339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c155478b-1ccc-4370-8b24-c4a2d22dda00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:42.337826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c155478b-1ccc-4370-8b24-c4a2d22dda00 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:42.348352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 755/1533] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:45.841087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e3f33f-ec05-458b-8c73-e4e624079b8a None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:45.841696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e3f33f-ec05-458b-8c73-e4e624079b8a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:45.841696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e3f33f-ec05-458b-8c73-e4e624079b8a None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:45.843154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 779/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:20:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:45.851623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b05608-74ed-40ff-9e91-479f494c79d0 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:45.852373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b05608-74ed-40ff-9e91-479f494c79d0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:45.852373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b05608-74ed-40ff-9e91-479f494c79d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:45.874527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b5b05608-74ed-40ff-9e91-479f494c79d0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:45.875803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b5b05608-74ed-40ff-9e91-479f494c79d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:45.885527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 756/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:20:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:20:46.448711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e3f33f-ec05-458b-8c73-e4e624079b8a None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:46.450883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-97e3f33f-ec05-458b-8c73-e4e624079b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:46.526801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:20:46.542278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e3f33f-ec05-458b-8c73-e4e624079b8a None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:46.546434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194af039-6487-4f03-868d-bdbd6d113949 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:46.547130 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194af039-6487-4f03-868d-bdbd6d113949 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:46.547538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194af039-6487-4f03-868d-bdbd6d113949 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:46.547904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194af039-6487-4f03-868d-bdbd6d113949 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:46.558116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194af039-6487-4f03-868d-bdbd6d113949 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:46.560245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194af039-6487-4f03-868d-bdbd6d113949 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:46.571188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 780/1536] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:48.506316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0acc6-e789-4072-a003-a4d52626dee3 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:48.507234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0acc6-e789-4072-a003-a4d52626dee3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:48.507234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0acc6-e789-4072-a003-a4d52626dee3 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:48.508220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 757/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sat Aug 27 02:20:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:20:48.521595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763cf1-5b76-4fad-94d3-b7fb4dd424a5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:48.521595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763cf1-5b76-4fad-94d3-b7fb4dd424a5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:48.522370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763cf1-5b76-4fad-94d3-b7fb4dd424a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:48.542057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-dc763cf1-5b76-4fad-94d3-b7fb4dd424a5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:20:48.543265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dc763cf1-5b76-4fad-94d3-b7fb4dd424a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:48.551537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 781/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Aug 27 02:20:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:20:49.117018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e0acc6-e789-4072-a003-a4d52626dee3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:20:49.117018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e0e0acc6-e789-4072-a003-a4d52626dee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:20:49.133744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e0acc6-e789-4072-a003-a4d52626dee3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:20:49.139733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e495be-72a0-4b3d-bfd5-d292bd13b3f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:20:49.140600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e495be-72a0-4b3d-bfd5-d292bd13b3f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:20:49.140962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e495be-72a0-4b3d-bfd5-d292bd13b3f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:20:49.141323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e495be-72a0-4b3d-bfd5-d292bd13b3f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:20:49.155018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e495be-72a0-4b3d-bfd5-d292bd13b3f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:20:49.156450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e495be-72a0-4b3d-bfd5-d292bd13b3f9 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:20:49.166128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 758/1539] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:20:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:20:58.537145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=11 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:20:59.161758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:17.320637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703d5c5-cb9e-4718-af6c-eec0d9c0d1cc None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.321822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703d5c5-cb9e-4718-af6c-eec0d9c0d1cc None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.322322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703d5c5-cb9e-4718-af6c-eec0d9c0d1cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.350123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d703d5c5-cb9e-4718-af6c-eec0d9c0d1cc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.352153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d703d5c5-cb9e-4718-af6c-eec0d9c0d1cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.360834 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 782/1540] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.369940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86699a-a7d9-4447-9594-55a064265b13 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.370526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86699a-a7d9-4447-9594-55a064265b13 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.370526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86699a-a7d9-4447-9594-55a064265b13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.406477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4a86699a-a7d9-4447-9594-55a064265b13 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.407717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4a86699a-a7d9-4447-9594-55a064265b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.417841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 759/1541] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.425609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d703d5c5-cb9e-4718-af6c-eec0d9c0d1cc None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:17.443626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d3e09-018d-4871-bf2f-2c29f6c2a770 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.443969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d3e09-018d-4871-bf2f-2c29f6c2a770 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.444185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d3e09-018d-4871-bf2f-2c29f6c2a770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.446647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d703d5c5-cb9e-4718-af6c-eec0d9c0d1cc None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:17.458187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:17.459122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.459614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.460134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.461156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:17.461675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:17.463494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:17.477147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b56d3e09-018d-4871-bf2f-2c29f6c2a770 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.479310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b56d3e09-018d-4871-bf2f-2c29f6c2a770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.487447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 783/1542] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:17.495827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:17.497444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 760/1543] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.501792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f62e7b-4eea-4c17-a555-905f9ae68c2c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.502072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f62e7b-4eea-4c17-a555-905f9ae68c2c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.502350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f62e7b-4eea-4c17-a555-905f9ae68c2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.515614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c70ebd-e773-4a51-9c58-d9078e202bef admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:17.531221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1ff968-e6da-4437-a711-df4f91131acb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:17.532289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1ff968-e6da-4437-a711-df4f91131acb admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.533009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1ff968-e6da-4437-a711-df4f91131acb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.536707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1ff968-e6da-4437-a711-df4f91131acb admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.537811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1ff968-e6da-4437-a711-df4f91131acb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:17.539406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-9f1ff968-e6da-4437-a711-df4f91131acb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:22:17.539406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:22:17.539406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:22:17.541965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1ff968-e6da-4437-a711-df4f91131acb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:17.543057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e3f62e7b-4eea-4c17-a555-905f9ae68c2c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.547542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e3f62e7b-4eea-4c17-a555-905f9ae68c2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.575618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 761/1544] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.576782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f744110-c70b-4153-b96c-ac6b4b1ec101 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.577073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f744110-c70b-4153-b96c-ac6b4b1ec101 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.577713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f744110-c70b-4153-b96c-ac6b4b1ec101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.626928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5f744110-c70b-4153-b96c-ac6b4b1ec101 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.628468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5f744110-c70b-4153-b96c-ac6b4b1ec101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.636203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 784/1545] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/projects => generated 369 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:22:17.637825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b40309-e91b-45b8-9589-2e97f053ca9f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.638109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b40309-e91b-45b8-9589-2e97f053ca9f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.638377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b40309-e91b-45b8-9589-2e97f053ca9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.647066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 762/1546] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.650257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9d1c65-8ab4-4ac3-a871-2927e2620059 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.650534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9d1c65-8ab4-4ac3-a871-2927e2620059 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.650807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9d1c65-8ab4-4ac3-a871-2927e2620059 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.669458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-12b40309-e91b-45b8-9589-2e97f053ca9f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.670669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-12b40309-e91b-45b8-9589-2e97f053ca9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.689347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ad9d1c65-8ab4-4ac3-a871-2927e2620059 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.693024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ad9d1c65-8ab4-4ac3-a871-2927e2620059 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.800225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 785/1547] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.801819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489e10c-ab6f-461a-98af-235111016681 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.802108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489e10c-ab6f-461a-98af-235111016681 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.802381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489e10c-ab6f-461a-98af-235111016681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.822601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 763/1548] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.823023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3489e10c-ab6f-461a-98af-235111016681 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.824400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9d1c65-8ab4-4ac3-a871-2927e2620059 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:17.824749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3489e10c-ab6f-461a-98af-235111016681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.833333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 786/1549] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.834162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3489e10c-ab6f-461a-98af-235111016681 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:17.836559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9d1c65-8ab4-4ac3-a871-2927e2620059 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:17.841697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fc6709-f655-40eb-b3b9-1fec94a9bf2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:17.842310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc6709-f655-40eb-b3b9-1fec94a9bf2f admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.842585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc6709-f655-40eb-b3b9-1fec94a9bf2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.842854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fc6709-f655-40eb-b3b9-1fec94a9bf2f admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.843502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fc6709-f655-40eb-b3b9-1fec94a9bf2f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:17.844407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3489e10c-ab6f-461a-98af-235111016681 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:17.844519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING py.warnings [None req-26fc6709-f655-40eb-b3b9-1fec94a9bf2f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:22:17.844519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: warnings.warn(msg) Aug 27 02:22:17.844519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: Aug 27 02:22:17.845753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fc6709-f655-40eb-b3b9-1fec94a9bf2f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:17.849006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4700971-9da6-43c6-af51-ca58ee5cd3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:17.849640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4700971-9da6-43c6-af51-ca58ee5cd3e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.849913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4700971-9da6-43c6-af51-ca58ee5cd3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.850174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4700971-9da6-43c6-af51-ca58ee5cd3e5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.850897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4700971-9da6-43c6-af51-ca58ee5cd3e5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:17.851195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4700971-9da6-43c6-af51-ca58ee5cd3e5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:17.852765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4700971-9da6-43c6-af51-ca58ee5cd3e5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:17.865443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 787/1550] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:17.867099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690db2a-6ffb-4cfe-a9f0-1c725e048a60 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.867614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690db2a-6ffb-4cfe-a9f0-1c725e048a60 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.867890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690db2a-6ffb-4cfe-a9f0-1c725e048a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.873563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 764/1551] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/users => generated 285 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:17.874699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9137483d-d115-4eea-9696-40b9b4508ccf None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.874982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9137483d-d115-4eea-9696-40b9b4508ccf None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.875249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9137483d-d115-4eea-9696-40b9b4508ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.887265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c690db2a-6ffb-4cfe-a9f0-1c725e048a60 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.888469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c690db2a-6ffb-4cfe-a9f0-1c725e048a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.894733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9137483d-d115-4eea-9696-40b9b4508ccf None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.895983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9137483d-d115-4eea-9696-40b9b4508ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.896361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 788/1552] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.898672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a396b-001a-4731-b0ec-e12030ec9eec None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.898953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a396b-001a-4731-b0ec-e12030ec9eec None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.899228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a396b-001a-4731-b0ec-e12030ec9eec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.904596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 765/1553] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.905811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7db8c-3d55-400a-a32f-ab9cd7733e25 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.906090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7db8c-3d55-400a-a32f-ab9cd7733e25 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.906358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7db8c-3d55-400a-a32f-ab9cd7733e25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.919765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2b7a396b-001a-4731-b0ec-e12030ec9eec None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.920947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2b7a396b-001a-4731-b0ec-e12030ec9eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.926723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5de7db8c-3d55-400a-a32f-ab9cd7733e25 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.927874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5de7db8c-3d55-400a-a32f-ab9cd7733e25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.929071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 789/1554] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.929883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7a396b-001a-4731-b0ec-e12030ec9eec None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:17.936932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 766/1555] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.938355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509324c2-3cc8-4fcf-bea1-9dc9eff4a41b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.938620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509324c2-3cc8-4fcf-bea1-9dc9eff4a41b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.938883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509324c2-3cc8-4fcf-bea1-9dc9eff4a41b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.940179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7a396b-001a-4731-b0ec-e12030ec9eec None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:17.945292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:17.945951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.946221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.946497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.947136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:17.947428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:17.949047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:17.963415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-509324c2-3cc8-4fcf-bea1-9dc9eff4a41b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:17.963941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 790/1556] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:17.964715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-509324c2-3cc8-4fcf-bea1-9dc9eff4a41b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:17.964867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:17.976590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 767/1557] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:17.977449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509324c2-3cc8-4fcf-bea1-9dc9eff4a41b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:17.980612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b4310-3e46-4529-9c9d-3337efeb86e2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:17.985441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1e672-0c53-4dae-bc79-eac32243a85e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:17.986045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1e672-0c53-4dae-bc79-eac32243a85e admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.986324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1e672-0c53-4dae-bc79-eac32243a85e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.986595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1e672-0c53-4dae-bc79-eac32243a85e admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.987229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1e672-0c53-4dae-bc79-eac32243a85e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:17.987632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509324c2-3cc8-4fcf-bea1-9dc9eff4a41b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:17.989136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1e672-0c53-4dae-bc79-eac32243a85e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:17.993065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8bd03-be62-44d4-b776-dc56665517d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:17.993685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8bd03-be62-44d4-b776-dc56665517d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:17.993953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8bd03-be62-44d4-b776-dc56665517d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:17.994218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8bd03-be62-44d4-b776-dc56665517d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:17.994845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca8bd03-be62-44d4-b776-dc56665517d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:17.996407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca8bd03-be62-44d4-b776-dc56665517d2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.006556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 768/1558] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:18.008154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0071520-4b1a-4998-9188-fc379c5cfc3c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.008504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0071520-4b1a-4998-9188-fc379c5cfc3c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.008872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0071520-4b1a-4998-9188-fc379c5cfc3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.022995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 791/1559] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:17 2022] POST /identity/v3/projects => generated 393 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.024293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1f7b08-da8b-4041-b25d-2cfb022a52ae None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.025099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1f7b08-da8b-4041-b25d-2cfb022a52ae None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.025410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1f7b08-da8b-4041-b25d-2cfb022a52ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.030443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-a0071520-4b1a-4998-9188-fc379c5cfc3c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:18.031734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a0071520-4b1a-4998-9188-fc379c5cfc3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:18.046237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6a1f7b08-da8b-4041-b25d-2cfb022a52ae None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:18.047433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6a1f7b08-da8b-4041-b25d-2cfb022a52ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:18.088657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:18.098475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 769/1560] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:18.102899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0071520-4b1a-4998-9188-fc379c5cfc3c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.112235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0071520-4b1a-4998-9188-fc379c5cfc3c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.114222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 792/1561] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:18.115468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125d11e-109f-4577-a588-19e64969d063 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.115747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125d11e-109f-4577-a588-19e64969d063 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.116006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125d11e-109f-4577-a588-19e64969d063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.118755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.119383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.121635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.121635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.121635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:18.121635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.121975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING py.warnings [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:22:18.121975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: warnings.warn(msg) Aug 27 02:22:18.121975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: Aug 27 02:22:18.122818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.135700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 770/1562] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:18.136044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b125d11e-109f-4577-a588-19e64969d063 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:18.136512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.137299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b125d11e-109f-4577-a588-19e64969d063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:18.148587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 793/1563] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:18.149424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b125d11e-109f-4577-a588-19e64969d063 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.159983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759fe028-a131-43ed-92be-23d75bd64054 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.162121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b125d11e-109f-4577-a588-19e64969d063 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.166014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.166626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.166911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.167231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.167869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.168497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.169150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.169447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.169718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.169869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.170351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:18.170648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.172298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.182226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:18.184984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 794/1564] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:18.186065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.197743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d29a7-fdc9-4b1c-bc69-858344e6846d admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.202941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.203554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.203825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.204087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.208440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 771/1565] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/projects => generated 373 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.208600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.208696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.208783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.218446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 795/1566] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:18.219285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.231747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b6e6c-ec30-4730-90bd-18ec2d47e9f0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.275352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ebce6-a08c-4a04-8d28-8c012e4bea4e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.282078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.282696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.282967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.283237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.283908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.285866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.292239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.292855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.293140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.293455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.294103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:18.294402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.296009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.311072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 772/1567] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/users => generated 297 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.311279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 796/1568] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:18.311279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.316010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.320803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a064de9-98d3-43ba-9195-69787042186c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.326210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72131c-edaa-4a61-bdde-e4167ef76923 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.326577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.327181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.327451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.327712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.328452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.330373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.331054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.331652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.331941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.332193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.332849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:18.333176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.334752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.347298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 773/1569] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:18.348139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.359678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6b29f5-36da-4ee2-b9fe-98c7ed653610 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.363868 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 797/1570] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/projects => generated 379 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.364672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.364820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.365373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.365696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.365963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.367316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:18.367685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.369631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.381472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 774/1571] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:18.382407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.393804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c929db4-5658-449b-8c58-ccbe60e2def3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.433722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6deb025-1d47-4bfd-9925-f05fd94a19c8 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.441666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.441666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.441666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.441666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.442132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.442951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.456788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.457440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.457719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.457984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.458652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.461036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.481709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 798/1572] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/projects => generated 385 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.481709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.485667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 775/1573] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/users => generated 287 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.486551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.492346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:18.497066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58da16b-afa3-444e-8182-feadeb25e51b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.539922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:18.546609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e9fa85-5cd2-4028-883f-0e2141b87372 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.551404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.552062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.552335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.552604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] PATH_INFO: `/v3/projects/e8e963bad9d04f4bb2a65eb42f86e188/users/37bb7d32464940aeaf2a144304a532d3/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.556324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.556930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.557231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.557500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.558175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.559774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.567767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8e963bad9d04f4bb2a65eb42f86e188, user_id=37bb7d32464940aeaf2a144304a532d3, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.568909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:22:18.568909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:22:18.568909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:22:18.570658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 776/1574] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:18.571227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.571447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.582998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d050a-c5d2-4212-92bd-e7137f6199b7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.597145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 799/1575] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:18 2022] PUT /identity/v3/projects/e8e963bad9d04f4bb2a65eb42f86e188/users/37bb7d32464940aeaf2a144304a532d3/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:18.597145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.649084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0bc3e2-6098-4526-b03d-10ba96096536 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.651065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.651726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.651955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.652223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.652864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.655005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.655641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.660136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.660136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.660136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.660136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.660136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.685717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 777/1576] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/projects => generated 373 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.686534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.692046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 800/1577] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/projects => generated 383 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.692927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.754710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95db375-528c-42bd-8b08-87b41a4d8b46 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.759927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.760533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15fae78-d0c6-420b-a559-dcb86d32c8e7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.760690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.760840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.761244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.762013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.763946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.765049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.765692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.765963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.766228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.766854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.768751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.790111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 778/1578] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.791226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.795368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 801/1579] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/projects => generated 403 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.795368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.798016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:18.801580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b665a29b-1072-4354-a8ac-3d3d14953b8f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.851298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:18.858726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a6229c-9e34-45d0-9b81-a6a02a7d4dc9 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.859918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ab42f-2bd6-4719-b1dd-7d0bf136272b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.860526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ab42f-2bd6-4719-b1dd-7d0bf136272b admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.860796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ab42f-2bd6-4719-b1dd-7d0bf136272b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.861058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ab42f-2bd6-4719-b1dd-7d0bf136272b admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.861721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ab42f-2bd6-4719-b1dd-7d0bf136272b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.863400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.863638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ab42f-2bd6-4719-b1dd-7d0bf136272b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.864003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.864275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.864633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.865429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.867539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.877794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 802/1580] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:18.878642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.888588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07359c9a-7c01-40b7-be2a-75b76a38804c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.889481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 779/1581] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/users => generated 293 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.890846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe412981-ae23-4098-9d02-95b79c224c48 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.891255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe412981-ae23-4098-9d02-95b79c224c48 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.891623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe412981-ae23-4098-9d02-95b79c224c48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.895661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.896261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.896531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.896812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.897780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.899428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.912376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 803/1582] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:18.912376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.921310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-fe412981-ae23-4098-9d02-95b79c224c48 None None] MFA Rules not processed for user `37bb7d32464940aeaf2a144304a532d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:18.921780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e828a82b-e813-46d5-a674-75be03383452 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.922754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fe412981-ae23-4098-9d02-95b79c224c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:18.926579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.927108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.927431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.927681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.928337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.930679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.953294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 804/1583] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/users => generated 287 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:18.954497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:18.964566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f779b1c-be12-4b30-a9fe-ab29814bb67d admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:18.970687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:18.971948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:18.972258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:18.972524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:18.973174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:18.975132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:18.991101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 780/1584] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:18.992589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe412981-ae23-4098-9d02-95b79c224c48 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.002602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 805/1585] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:18 2022] POST /identity/v3/users => generated 292 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:19.003463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.006415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe412981-ae23-4098-9d02-95b79c224c48 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.012410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.013189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.013468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.013572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c7d4a7-70c3-48c9-a5c6-f9ae347f0afb admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.013771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.014453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.016060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.018231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.018822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.019093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.019358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.019977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.021894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.029825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 781/1586] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.029825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.037582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c814814-ee0f-46ee-ae52-4cad8eaaa60b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.042676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.043273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.043548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.043794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.044433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.046331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.046462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 806/1587] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:19 2022] POST /identity/v3/users => generated 302 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:19.047078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.056525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec7241-1647-4fd9-af53-8bde667f0930 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.062054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 782/1588] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.062054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.062269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.062269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.062269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.062542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.062955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.064666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.074408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e612d5-8fed-4213-9bc1-9e40c0809aef admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.079831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 807/1589] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.080942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.083589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.084306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.084578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.084846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] PATH_INFO: `/v3/projects/880ac4003248441c809a76cfd1b221eb/users/d945d512ae034ec08838d0a3afde2429/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.092660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedf2a70-f2ad-42ff-b80d-348987c7673a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.097157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.097786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.098054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.098332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.098962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.100352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=880ac4003248441c809a76cfd1b221eb, user_id=d945d512ae034ec08838d0a3afde2429, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.100573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.103034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.111797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 808/1590] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.112595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.122659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f775a8e5-5c37-4e49-806d-84bedd83bbb5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.124157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 783/1591] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:19 2022] PUT /identity/v3/projects/880ac4003248441c809a76cfd1b221eb/users/d945d512ae034ec08838d0a3afde2429/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:19.125326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.184438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.184929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.185223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.185487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.186146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.188002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.193856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8d8454-1811-4b46-9364-a34cf958ff08 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.200399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.201240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.201540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.201830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.202534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.204269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.209058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:19.219434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 784/1592] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.219434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.220322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 809/1593] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:19 2022] POST /identity/v3/projects => generated 373 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:19.221390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.228313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eb03cb-0556-403d-a107-1994dcfc3ca8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.248716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:19.284926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa4a602-47aa-45fa-b1ef-41b2467b335a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.288867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.289648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.289922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.290221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.290738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.290877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.291354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.291630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.292050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.292411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.292705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.294320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.303474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 785/1594] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.304403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.307941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 810/1595] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.307941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.317193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef84c019-9809-47f4-981a-cc768d1abc9e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.320441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9617e29-7d06-463f-88be-8e7f63cefeda admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.323082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.323712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.323983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.324557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.325265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.326139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a7b094-bcc4-48eb-9ae5-302cd6ad2799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.326815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a7b094-bcc4-48eb-9ae5-302cd6ad2799 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.326936 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.327096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a7b094-bcc4-48eb-9ae5-302cd6ad2799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.327381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a7b094-bcc4-48eb-9ae5-302cd6ad2799 admin admin] PATH_INFO: `/v3/projects/6904bf36e551422b862effe3e98b85f6/users/780f06ac3efb41b188805de61d79b372/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.337784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 811/1596] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.338444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.344477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a7b094-bcc4-48eb-9ae5-302cd6ad2799 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6904bf36e551422b862effe3e98b85f6, user_id=780f06ac3efb41b188805de61d79b372, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.347298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a7b094-bcc4-48eb-9ae5-302cd6ad2799 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.348637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736c5c5f-2e2a-4270-a149-ca91db21cdcb admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.354875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.355729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.356011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.356330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.357280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.358756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.361802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 786/1597] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:19 2022] PUT /identity/v3/projects/6904bf36e551422b862effe3e98b85f6/users/780f06ac3efb41b188805de61d79b372/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:19.371823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93cdbdb-3dc3-4891-b0e0-3616c93a96f5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.371823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93cdbdb-3dc3-4891-b0e0-3616c93a96f5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.371823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93cdbdb-3dc3-4891-b0e0-3616c93a96f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.378393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 812/1598] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.379312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.383583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e93cdbdb-3dc3-4891-b0e0-3616c93a96f5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:19.384787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e93cdbdb-3dc3-4891-b0e0-3616c93a96f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:19.390265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc79a31-4266-4f80-8192-c914edb1aa7f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.447113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 787/1599] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:19.447625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.447842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93cdbdb-3dc3-4891-b0e0-3616c93a96f5 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.448321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.448601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.448871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.449858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.451616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.458863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93cdbdb-3dc3-4891-b0e0-3616c93a96f5 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.463426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 813/1600] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.464228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.464994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.465707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.465985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.466269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.466911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.468862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.479202 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f67868-f37d-4fef-a3c7-7ac4da36f811 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.485896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.486573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.487078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.487495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] PATH_INFO: `/v3/projects/f7ef4b44a3f94d599eacce269b06594c/users/55640f750d1b49be9c3f79501a63c1ab/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.492367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 788/1601] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:19 2022] POST /identity/v3/users => generated 293 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:19.493222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.506268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727baeb7-66dc-45e5-b8a1-c5c6acfa2770 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.506653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7ef4b44a3f94d599eacce269b06594c, user_id=55640f750d1b49be9c3f79501a63c1ab, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.509771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.512152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f4b5a0-fa07-446f-bef2-522fe6f70d61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.513058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f4b5a0-fa07-446f-bef2-522fe6f70d61 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.513058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f4b5a0-fa07-446f-bef2-522fe6f70d61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.513356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f4b5a0-fa07-446f-bef2-522fe6f70d61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.513969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f4b5a0-fa07-446f-bef2-522fe6f70d61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.515692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f4b5a0-fa07-446f-bef2-522fe6f70d61 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.524521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 814/1602] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:19 2022] PUT /identity/v3/projects/f7ef4b44a3f94d599eacce269b06594c/users/55640f750d1b49be9c3f79501a63c1ab/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:19.525573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.530234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 789/1603] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.531529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3535cc-9cbc-48c1-9154-24edcee45bfb None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.531812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3535cc-9cbc-48c1-9154-24edcee45bfb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.532091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3535cc-9cbc-48c1-9154-24edcee45bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.560642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2f3535cc-9cbc-48c1-9154-24edcee45bfb None None] MFA Rules not processed for user `780f06ac3efb41b188805de61d79b372`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:19.561978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2f3535cc-9cbc-48c1-9154-24edcee45bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:19.574593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:19.594595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44b905-ef34-43e1-bdd7-b47bed8bcaae admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.599703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.600443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.600970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.601534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] PATH_INFO: `/v3/projects/5d5ec8450c0941d18d63558baab54737/users/26a86351583d49d090b3db09f127d49f/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.616452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d5ec8450c0941d18d63558baab54737, user_id=26a86351583d49d090b3db09f127d49f, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.619257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.625003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:19.632482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 790/1604] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:19.633597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3535cc-9cbc-48c1-9154-24edcee45bfb None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.637619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 815/1605] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:19 2022] PUT /identity/v3/projects/5d5ec8450c0941d18d63558baab54737/users/26a86351583d49d090b3db09f127d49f/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:19.637619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.646871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3535cc-9cbc-48c1-9154-24edcee45bfb None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.690631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857a419a-e9f4-477f-a4a5-4d543f54de8b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.694645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.695265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.695540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.695808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.708878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.709504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.709782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.710146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] PATH_INFO: `/v3/projects/458b9db3913441959977889262b2236c/users/d639b73e786f40cda27c02d580c7c9ec/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.728036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=458b9db3913441959977889262b2236c, user_id=d639b73e786f40cda27c02d580c7c9ec, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.730922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.744387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 791/1606] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:19 2022] PUT /identity/v3/projects/458b9db3913441959977889262b2236c/users/d639b73e786f40cda27c02d580c7c9ec/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:19.745210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.764663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.766167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.776648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 816/1607] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:19.777429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.786534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc1c9be-afa0-45b8-842d-651f7587f959 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.822746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4d9960-c815-4279-80c7-4fe20f164ba6 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:19.830092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.831385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.831670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.832314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] PATH_INFO: `/v3/projects/5c41fc70b9774df08ee9b819edc12ba8/users/7aa09b0963d04b588386f1c0a96e2128/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.851308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8582d1-bacb-4a99-8239-322f7f657549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:19.851938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8582d1-bacb-4a99-8239-322f7f657549 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.852240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8582d1-bacb-4a99-8239-322f7f657549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.852474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8582d1-bacb-4a99-8239-322f7f657549 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.853145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8582d1-bacb-4a99-8239-322f7f657549 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.854269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c41fc70b9774df08ee9b819edc12ba8, user_id=7aa09b0963d04b588386f1c0a96e2128, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:19.854733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8582d1-bacb-4a99-8239-322f7f657549 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.856909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:19.867826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 817/1608] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:19.869366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811d2d8e-d0f4-4c0f-83fb-4bd19d7312df None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:19.869781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811d2d8e-d0f4-4c0f-83fb-4bd19d7312df None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:19.870198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811d2d8e-d0f4-4c0f-83fb-4bd19d7312df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:19.874237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 792/1609] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:19 2022] PUT /identity/v3/projects/5c41fc70b9774df08ee9b819edc12ba8/users/7aa09b0963d04b588386f1c0a96e2128/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:19.874237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:19.907278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-811d2d8e-d0f4-4c0f-83fb-4bd19d7312df None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:19.908648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-811d2d8e-d0f4-4c0f-83fb-4bd19d7312df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:19.920851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:19.988845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7211da-00fb-4c39-90fb-f21423a4028a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.000144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1f4e20-f2e9-489e-bde0-7cf5bd6e821d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.000144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f4e20-f2e9-489e-bde0-7cf5bd6e821d admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.000144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f4e20-f2e9-489e-bde0-7cf5bd6e821d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.001240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f4e20-f2e9-489e-bde0-7cf5bd6e821d admin admin] PATH_INFO: `/v3/projects/772942aea6ee43ff8146ab63c261c275/users/af3661508dea49dfa7301ae6c372b7fd/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.014839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:20.031565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1f4e20-f2e9-489e-bde0-7cf5bd6e821d admin admin] RBAC: Authorizing `identity:create_grant(project_id=772942aea6ee43ff8146ab63c261c275, user_id=af3661508dea49dfa7301ae6c372b7fd, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.032136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1f4e20-f2e9-489e-bde0-7cf5bd6e821d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.040916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 818/1610] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:20.040916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811d2d8e-d0f4-4c0f-83fb-4bd19d7312df None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.058653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811d2d8e-d0f4-4c0f-83fb-4bd19d7312df None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.061256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 793/1611] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:19 2022] PUT /identity/v3/projects/772942aea6ee43ff8146ab63c261c275/users/af3661508dea49dfa7301ae6c372b7fd/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:20.062881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204fd6da-768f-4edc-8d07-c7258742c122 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.063251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204fd6da-768f-4edc-8d07-c7258742c122 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.063603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204fd6da-768f-4edc-8d07-c7258742c122 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.088421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-204fd6da-768f-4edc-8d07-c7258742c122 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:20.089814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-204fd6da-768f-4edc-8d07-c7258742c122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:20.143002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c909b9cd-d8b0-49e5-881d-9ca784ac1c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.144253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c909b9cd-d8b0-49e5-881d-9ca784ac1c8a admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.145184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c909b9cd-d8b0-49e5-881d-9ca784ac1c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.145587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c909b9cd-d8b0-49e5-881d-9ca784ac1c8a admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.146962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c909b9cd-d8b0-49e5-881d-9ca784ac1c8a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.150225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c909b9cd-d8b0-49e5-881d-9ca784ac1c8a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.203132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 819/1612] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:20.208164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 794/1613] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:20.209799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204fd6da-768f-4edc-8d07-c7258742c122 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.210439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3351e4-4f70-4403-a4ee-084c79e82bfb None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.210880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3351e4-4f70-4403-a4ee-084c79e82bfb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.211395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3351e4-4f70-4403-a4ee-084c79e82bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.246655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204fd6da-768f-4edc-8d07-c7258742c122 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.263068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-fa3351e4-4f70-4403-a4ee-084c79e82bfb None None] MFA Rules not processed for user `d639b73e786f40cda27c02d580c7c9ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:20.265007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fa3351e4-4f70-4403-a4ee-084c79e82bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:20.336970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.337737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.338035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.338309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] PATH_INFO: `/v3/projects/7fd6329902534d67bb14510a4c27bdd1/users/737cb56f94494015ab44fe6dc4496d9a/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.363748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fd6329902534d67bb14510a4c27bdd1, user_id=737cb56f94494015ab44fe6dc4496d9a, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.367507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.371501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 820/1614] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:20.373338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa8732-da3b-4901-968b-ca305f1c94d0 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.373338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa8732-da3b-4901-968b-ca305f1c94d0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.373820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa8732-da3b-4901-968b-ca305f1c94d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.391175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 795/1615] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:20 2022] PUT /identity/v3/projects/7fd6329902534d67bb14510a4c27bdd1/users/737cb56f94494015ab44fe6dc4496d9a/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:20.391175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.391175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:20.412630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-40aa8732-da3b-4901-968b-ca305f1c94d0 None None] MFA Rules not processed for user `7aa09b0963d04b588386f1c0a96e2128`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:20.414577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-40aa8732-da3b-4901-968b-ca305f1c94d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:20.438344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:20.475030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21ab4f4-8214-4ed6-89bd-ba6cb547db17 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.481038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.481751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.482084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.482292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.505335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 821/1616] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:20.507547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1725246-8ceb-4119-a9a9-451d9386e94c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.508048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1725246-8ceb-4119-a9a9-451d9386e94c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.508310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1725246-8ceb-4119-a9a9-451d9386e94c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.553666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f1725246-8ceb-4119-a9a9-451d9386e94c None None] MFA Rules not processed for user `af3661508dea49dfa7301ae6c372b7fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:20.557534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f1725246-8ceb-4119-a9a9-451d9386e94c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:20.559971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:20.651346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.658148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.661222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 822/1617] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:20.666753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1725246-8ceb-4119-a9a9-451d9386e94c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.681109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 796/1618] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:20.686095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.694182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1725246-8ceb-4119-a9a9-451d9386e94c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.705378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b280685-8028-4d53-9660-9a3214e30229 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.708017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.708642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.708946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.711209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.712070 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.717031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.721044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.721749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.722526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.722526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.747206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.748962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.765527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 797/1619] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:20.766515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.780438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2f3e7e-15a2-4442-bee0-8624fde9b401 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.786578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e8102e-7f58-4c48-8d82-68cf29881295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.791808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 823/1620] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/users => generated 295 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:20.792058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e8102e-7f58-4c48-8d82-68cf29881295 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.792058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e8102e-7f58-4c48-8d82-68cf29881295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.792058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e8102e-7f58-4c48-8d82-68cf29881295 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.792490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.792629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e8102e-7f58-4c48-8d82-68cf29881295 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.792629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e8102e-7f58-4c48-8d82-68cf29881295 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.823621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10e1a80-9acf-4fe6-baad-87fc517e6fac admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:20.838882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:20.840091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.840637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.841231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.843026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:20.846577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 798/1621] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/projects => generated 373 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:20.846982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:20.855757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b40a58-ba83-46c3-b87a-dcc306d14887 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:20.856657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b40a58-ba83-46c3-b87a-dcc306d14887 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:20.857419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b40a58-ba83-46c3-b87a-dcc306d14887 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:20.868422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 799/1622] 10.222.0.52 () {54 vars in 962 bytes} [Sat Aug 27 02:22:20 2022] GET /identity => generated 270 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:22:20.873205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b40a58-ba83-46c3-b87a-dcc306d14887 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.892757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 824/1623] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:20 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:20.892757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:20.923719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3888e29e-f7fa-455a-96d8-7b9f03c02018 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.047851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:21.049091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.049473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.049908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.051306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:21.054790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:21.086564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 825/1624] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:20 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:21.087802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:21.111513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafa4f38-591d-415e-adb7-82f9b2b471f3 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.122242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb697914-a3e4-45ce-b53e-a0dab69143ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:21.123632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb697914-a3e4-45ce-b53e-a0dab69143ba admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.124098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb697914-a3e4-45ce-b53e-a0dab69143ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.124618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb697914-a3e4-45ce-b53e-a0dab69143ba admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.125805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb697914-a3e4-45ce-b53e-a0dab69143ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:21.129553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb697914-a3e4-45ce-b53e-a0dab69143ba admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:21.146939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b40a58-ba83-46c3-b87a-dcc306d14887 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.162729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:21.164136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.165632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.169187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.170576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:21.177275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:21.181027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:21.196006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 826/1625] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:21 2022] POST /identity/v3/users => generated 293 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:21.197040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd3a2c-9b41-4c11-b9f2-628cb8fc5b1f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.197942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd3a2c-9b41-4c11-b9f2-628cb8fc5b1f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.198518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd3a2c-9b41-4c11-b9f2-628cb8fc5b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.307056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 800/1626] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:20 2022] POST /identity/v3/projects => generated 405 bytes in 434 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:21.309310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:21.363739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-e7bd3a2c-9b41-4c11-b9f2-628cb8fc5b1f None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:21.365878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e7bd3a2c-9b41-4c11-b9f2-628cb8fc5b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:21.445509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e4edfc-f27a-4d4d-b568-1317d2cad9e5 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.462698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:21.463397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.463683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.463954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] PATH_INFO: `/v3/projects/7fd6329902534d67bb14510a4c27bdd1/users/737cb56f94494015ab44fe6dc4496d9a/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.466700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fd6329902534d67bb14510a4c27bdd1, user_id=737cb56f94494015ab44fe6dc4496d9a, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:21.469615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:21.503651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 827/1627] 10.222.0.52 () {58 vars in 1059 bytes} [Sat Aug 27 02:22:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:21.505028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd3a2c-9b41-4c11-b9f2-628cb8fc5b1f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:21.507827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 801/1628] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:21 2022] PUT /identity/v3/projects/7fd6329902534d67bb14510a4c27bdd1/users/737cb56f94494015ab44fe6dc4496d9a/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:21.508751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:21.525791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:21.527326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd3a2c-9b41-4c11-b9f2-628cb8fc5b1f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.609644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f14037a-59ee-4ae7-897a-0b5df197aadc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.615174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:21.615957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.616447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.617654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.618507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:21.620176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:21.634294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c6a331-793a-488e-a943-09a98676eb28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:21.635269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c6a331-793a-488e-a943-09a98676eb28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.635690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c6a331-793a-488e-a943-09a98676eb28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.636122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c6a331-793a-488e-a943-09a98676eb28 admin admin] PATH_INFO: `/v3/projects/37f8fe6f87be453baaaca4c47bc6a0f7/users/d38ffa2630af41ff848e98418e551057/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.649877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 802/1629] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:21 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:21.651040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:21.662647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0922943-3ea6-406b-a4d9-8db3678e51ee admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.663462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c6a331-793a-488e-a943-09a98676eb28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37f8fe6f87be453baaaca4c47bc6a0f7, user_id=d38ffa2630af41ff848e98418e551057, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:21.668941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:21.669857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.670233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.670595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.671112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c6a331-793a-488e-a943-09a98676eb28 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:21.671425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:21.673700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:21.693732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 828/1630] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:21 2022] PUT /identity/v3/projects/37f8fe6f87be453baaaca4c47bc6a0f7/users/d38ffa2630af41ff848e98418e551057/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:21.696997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427b2ef-b690-4a1c-9ca1-83175ec12dd8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:21.698348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427b2ef-b690-4a1c-9ca1-83175ec12dd8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:21.698937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427b2ef-b690-4a1c-9ca1-83175ec12dd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:21.703529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 803/1631] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:21 2022] POST /identity/v3/users => generated 309 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:21.706115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:21.764282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5427b2ef-b690-4a1c-9ca1-83175ec12dd8 None None] MFA Rules not processed for user `737cb56f94494015ab44fe6dc4496d9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:21.766274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5427b2ef-b690-4a1c-9ca1-83175ec12dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:21.896904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2c99a-5a56-4707-a1b4-7ed4ecedd288 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:21.911276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:21.990476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 829/1632] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:21 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:22.008451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5427b2ef-b690-4a1c-9ca1-83175ec12dd8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.059477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.061057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.061623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.062053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.090748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:22.155830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5427b2ef-b690-4a1c-9ca1-83175ec12dd8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.161390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.162197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.162574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.162938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] PATH_INFO: `/v3/projects/c169fcfeeaf245fc81ba9ecbbc73363a/users/771ad70b7a41447fae5d1059d18d7466/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.186409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c169fcfeeaf245fc81ba9ecbbc73363a, user_id=771ad70b7a41447fae5d1059d18d7466, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.190385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.200147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.207383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.231408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 830/1633] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:21 2022] PUT /identity/v3/projects/c169fcfeeaf245fc81ba9ecbbc73363a/users/771ad70b7a41447fae5d1059d18d7466/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:22.234820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.252126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 804/1634] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 545 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:22.255637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.297052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620111dd-edb6-45b2-abbd-27b443760016 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.396250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.397033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.397431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.398357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.399349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.401360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.417190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 805/1635] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:22.418271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.436569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f683f3-4602-4d14-8c3c-35432908b08c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.446716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.448268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.449002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.450180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.451933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.455272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.490131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 831/1636] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:22.495308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.497462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69391829-cf0e-4f28-bf80-c8fa00de9d37 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.502048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.502680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.502963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.503232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.514006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd89a0e-36ba-49b9-a14b-e5ec10327556 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.523188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.524538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.525382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.526084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.527691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.530883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.564641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 832/1637] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:22.567697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.596371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee762d6-d498-4581-b9b4-56d3a81571b9 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.620995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.624189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.624822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.625194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] PATH_INFO: `/v3/projects/f955963f181a4e8f9274cdc63dca5880/users/c9d4e0014f8d40899201f028d5e88042/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.663491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.666026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.671387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f955963f181a4e8f9274cdc63dca5880, user_id=c9d4e0014f8d40899201f028d5e88042, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.676739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.694848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 806/1638] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:22 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:22.697605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.705838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 833/1639] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:22 2022] PUT /identity/v3/projects/f955963f181a4e8f9274cdc63dca5880/users/c9d4e0014f8d40899201f028d5e88042/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:22.707088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.730314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:22.742907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1440cc0-7864-4a52-8078-34dcfbcaf279 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.804886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:22.892108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.892800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.892986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfe3b2-a772-488f-ac3c-0547cc829d21 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:22.893221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.893399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] PATH_INFO: `/v3/projects/37f8fe6f87be453baaaca4c47bc6a0f7/users/d38ffa2630af41ff848e98418e551057/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.896445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37f8fe6f87be453baaaca4c47bc6a0f7, user_id=d38ffa2630af41ff848e98418e551057, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.899957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.906541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:22.908347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:22.909751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:22.912455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] PATH_INFO: `/v3/projects/c169fcfeeaf245fc81ba9ecbbc73363a/users/771ad70b7a41447fae5d1059d18d7466/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:22.924925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c169fcfeeaf245fc81ba9ecbbc73363a, user_id=771ad70b7a41447fae5d1059d18d7466, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:22.946912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 807/1640] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:22 2022] PUT /identity/v3/projects/37f8fe6f87be453baaaca4c47bc6a0f7/users/d38ffa2630af41ff848e98418e551057/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:22.947430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:22.947741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:22.959638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 834/1641] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:22 2022] PUT /identity/v3/projects/c169fcfeeaf245fc81ba9ecbbc73363a/users/771ad70b7a41447fae5d1059d18d7466/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:22.961799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:23.090816 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99e4c3-e1b6-401b-8b3a-c208bafe2fd0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:23.184653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac9e3b7-ee14-4d43-ade4-5e99c5415852 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:23.198179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:23.200384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.202311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.203052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.222308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a114c5-3f04-4d11-9245-a2a06123087f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:23.223029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a114c5-3f04-4d11-9245-a2a06123087f admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.223419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a114c5-3f04-4d11-9245-a2a06123087f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.223692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a114c5-3f04-4d11-9245-a2a06123087f admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.225888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a114c5-3f04-4d11-9245-a2a06123087f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:23.227846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a114c5-3f04-4d11-9245-a2a06123087f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:23.259469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 808/1642] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:22 2022] GET /identity/v3/roles => generated 1420 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:23.265730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d5e4d1-7700-4a9f-8909-8e5e3c8f74e2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.265999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d5e4d1-7700-4a9f-8909-8e5e3c8f74e2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.266282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d5e4d1-7700-4a9f-8909-8e5e3c8f74e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.334632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f0d5e4d1-7700-4a9f-8909-8e5e3c8f74e2 None None] MFA Rules not processed for user `d38ffa2630af41ff848e98418e551057`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:23.336635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f0d5e4d1-7700-4a9f-8909-8e5e3c8f74e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:23.378301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:23.409977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:23.414129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:23.429148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 809/1643] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:23 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:23.430483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ed0089-cc25-4d8e-8dc1-37b451d5a981 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.430903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ed0089-cc25-4d8e-8dc1-37b451d5a981 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.431172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ed0089-cc25-4d8e-8dc1-37b451d5a981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.440725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 835/1644] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:22 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:23.443687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:23.468237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d885bb09-1775-414f-aada-cd667e961081 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:23.472126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-73ed0089-cc25-4d8e-8dc1-37b451d5a981 None None] MFA Rules not processed for user `771ad70b7a41447fae5d1059d18d7466`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:23.472683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-73ed0089-cc25-4d8e-8dc1-37b451d5a981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:23.481751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bb4ffc-c427-4c80-bf33-4fe1378a5d2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:23.483560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bb4ffc-c427-4c80-bf33-4fe1378a5d2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.484826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bb4ffc-c427-4c80-bf33-4fe1378a5d2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.486040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bb4ffc-c427-4c80-bf33-4fe1378a5d2c admin admin] PATH_INFO: `/v3/projects/f955963f181a4e8f9274cdc63dca5880/users/c9d4e0014f8d40899201f028d5e88042/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.490158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bb4ffc-c427-4c80-bf33-4fe1378a5d2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f955963f181a4e8f9274cdc63dca5880, user_id=c9d4e0014f8d40899201f028d5e88042, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:23.498322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bb4ffc-c427-4c80-bf33-4fe1378a5d2c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:23.527562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 836/1645] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:23 2022] PUT /identity/v3/projects/f955963f181a4e8f9274cdc63dca5880/users/c9d4e0014f8d40899201f028d5e88042/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:23.531130 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f433abbf-ee1a-4e80-b388-50b836a42d91 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.531939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f433abbf-ee1a-4e80-b388-50b836a42d91 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.533770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f433abbf-ee1a-4e80-b388-50b836a42d91 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.546733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 837/1646] 10.222.0.52 () {54 vars in 912 bytes} [Sat Aug 27 02:22:23 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:22:23.549868 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f433abbf-ee1a-4e80-b388-50b836a42d91 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:23.593424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 810/1647] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:23 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:23.594717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125db3bc-6860-490d-964b-1de83f9f31a2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.595172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125db3bc-6860-490d-964b-1de83f9f31a2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.595450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125db3bc-6860-490d-964b-1de83f9f31a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.638138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-125db3bc-6860-490d-964b-1de83f9f31a2 None None] MFA Rules not processed for user `c9d4e0014f8d40899201f028d5e88042`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:23.641521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-125db3bc-6860-490d-964b-1de83f9f31a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:23.718343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f433abbf-ee1a-4e80-b388-50b836a42d91 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:23.740891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:23.742442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.742845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.743738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.775838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:23.835694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 811/1648] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:23 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:23.841209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10488ce4-8bd3-4086-aabb-1be977787ee7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:23.841658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10488ce4-8bd3-4086-aabb-1be977787ee7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:23.842103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10488ce4-8bd3-4086-aabb-1be977787ee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:23.845708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:23.848676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:23.864274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 838/1649] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:23 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:23.924260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-10488ce4-8bd3-4086-aabb-1be977787ee7 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:23.926953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-10488ce4-8bd3-4086-aabb-1be977787ee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:24.048099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:24.056708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:24.070536 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5760dd66-b7a6-43b4-bb6f-e8338d66ccca service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:24.087895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:24.088533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:24.092145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:24.094136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:24.176408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 812/1650] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:22:23 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:24.276347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:24.280110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:24.306817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 839/1651] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:24 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:22:24.488256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10488ce4-8bd3-4086-aabb-1be977787ee7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:24.528567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10488ce4-8bd3-4086-aabb-1be977787ee7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:24.561189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:24.563441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:24.564929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:24.565420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:24.632010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:24.636534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:24.665501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 813/1652] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:24 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:24.826483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:24.842321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab7d18c-0d8a-4a3e-9ca1-089e5f3dd232 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:24.852787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5882f0-d641-4d02-b399-e6fc4912a849 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:24.853504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5882f0-d641-4d02-b399-e6fc4912a849 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:24.853809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5882f0-d641-4d02-b399-e6fc4912a849 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:24.854080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5882f0-d641-4d02-b399-e6fc4912a849 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:24.881867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5882f0-d641-4d02-b399-e6fc4912a849 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:24.894219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5882f0-d641-4d02-b399-e6fc4912a849 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:24.967099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 840/1653] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:26.445883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:26.625477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9cd263-0e21-4370-8c62-84a3c3162650 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:26.655836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:26.656516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:26.663958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:26.663958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] PATH_INFO: `/v3/users/af3661508dea49dfa7301ae6c372b7fd` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:26.670927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af3661508dea49dfa7301ae6c372b7fd)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:26.682901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:26.799150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af3661508dea49dfa7301ae6c372b7fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:26.824379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af3661508dea49dfa7301ae6c372b7fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:26.850868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af3661508dea49dfa7301ae6c372b7fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:26.852370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af3661508dea49dfa7301ae6c372b7fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:26.853537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-db158b4f-7cb1-4889-94b0-fdf07929fe39 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af3661508dea49dfa7301ae6c372b7fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:26.898136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 814/1654] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:22:26 2022] DELETE /identity/v3/users/af3661508dea49dfa7301ae6c372b7fd => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:26.900353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:26.901342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:26.901816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:26.965154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:26.968619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:27.084924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5314f55d-9df8-41d9-9355-713855c43c4e None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:27.085652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5314f55d-9df8-41d9-9355-713855c43c4e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:27.086109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5314f55d-9df8-41d9-9355-713855c43c4e None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:27.087673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 815/1655] 10.222.0.52 () {54 vars in 962 bytes} [Sat Aug 27 02:22:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:22:27.108624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4b022c-7867-488c-9069-ac40c6f88742 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:27.108999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4b022c-7867-488c-9069-ac40c6f88742 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:27.109386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4b022c-7867-488c-9069-ac40c6f88742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:27.140377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ca4b022c-7867-488c-9069-ac40c6f88742 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:27.147591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ca4b022c-7867-488c-9069-ac40c6f88742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:27.202861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 841/1656] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:27.228340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:27.230068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:27.279380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:27.391161 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:27.461815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 816/1657] 10.222.0.52 () {58 vars in 1059 bytes} [Sat Aug 27 02:22:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:27.473545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed13be53-f41e-4828-857b-27b9e12fbc7c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:27.483276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc8f35a-805c-43dc-bed0-c3b9cb863050 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:27.483566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc8f35a-805c-43dc-bed0-c3b9cb863050 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:27.483844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc8f35a-805c-43dc-bed0-c3b9cb863050 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:27.490582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 817/1658] 10.222.0.52 () {54 vars in 912 bytes} [Sat Aug 27 02:22:27 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:22:27.494422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:27.494763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:27.495044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:27.495301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:27.498345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:27.499055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:27.502994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:27.555776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:27.559488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:27.568630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:27.575263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:27.594168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 842/1659] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:27.632202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 818/1660] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:22:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Aug 27 02:22:27.694816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:27.696738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:27.709258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:27.712386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:27.751230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca32be06-ed6d-4fb8-9826-03af08469ec5 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:27.770828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:27.771573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:27.771853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:27.772410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] PATH_INFO: `/v3/users/d639b73e786f40cda27c02d580c7c9ec` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:27.779326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d639b73e786f40cda27c02d580c7c9ec)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:27.800653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:27.935088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d39e3d-0bc4-42f8-a895-120c46363e0e service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:27.939632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd639b73e786f40cda27c02d580c7c9ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:27.960741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210132ae-01f2-425c-aa57-48e9abeead00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:27.965251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210132ae-01f2-425c-aa57-48e9abeead00 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:27.965542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210132ae-01f2-425c-aa57-48e9abeead00 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:27.965812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210132ae-01f2-425c-aa57-48e9abeead00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:27.973731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd639b73e786f40cda27c02d580c7c9ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:27.988911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210132ae-01f2-425c-aa57-48e9abeead00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:27.991104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd639b73e786f40cda27c02d580c7c9ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:27.991823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd639b73e786f40cda27c02d580c7c9ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:27.992719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd639b73e786f40cda27c02d580c7c9ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:27.998206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210132ae-01f2-425c-aa57-48e9abeead00 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:28.053473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 843/1661] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:28.061069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 819/1662] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:22:27 2022] DELETE /identity/v3/users/d639b73e786f40cda27c02d580c7c9ec => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:28.062915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44728440-1182-4aae-95f8-998491932961 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:28.063271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44728440-1182-4aae-95f8-998491932961 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:28.063612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44728440-1182-4aae-95f8-998491932961 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:28.118451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-44728440-1182-4aae-95f8-998491932961 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:28.123251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-44728440-1182-4aae-95f8-998491932961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:28.381089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 844/1663] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:28.509161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:28.511562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:28.767322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afc4a90-8097-4cac-9904-6451d2c0aa48 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:28.779981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:28.783466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:28.783938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:28.784999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:28.805383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:28.807013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:28.827412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 820/1664] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:29.041500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44728440-1182-4aae-95f8-998491932961 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:29.045075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-44728440-1182-4aae-95f8-998491932961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:29.068737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44728440-1182-4aae-95f8-998491932961 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:29.085571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:29.089066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:29.089066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:29.089066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] PATH_INFO: `/v3/projects/772942aea6ee43ff8146ab63c261c275` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:29.091191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] RBAC: Authorizing `identity:delete_project(project_id=772942aea6ee43ff8146ab63c261c275)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:29.093452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:29.171647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '772942aea6ee43ff8146ab63c261c275', 'request_id': 'req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.196470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '772942aea6ee43ff8146ab63c261c275', 'request_id': 'req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.215568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '772942aea6ee43ff8146ab63c261c275', 'request_id': 'req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.238665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] The token cache is being invalidate because project 772942aea6ee43ff8146ab63c261c275 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:22:29.239437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.240480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b466ffb0-c4a5-4f96-a1e9-1a393a5ad99f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.243382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 845/1665] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:22:29 2022] DELETE /identity/v3/projects/772942aea6ee43ff8146ab63c261c275 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:29.247709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:29.265779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a9bdd6-6329-47e2-bff7-f213b30dfadc service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:29.363076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:29.446347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:29.447062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:29.447613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:29.447613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] PATH_INFO: `/v3/users/c9d4e0014f8d40899201f028d5e88042` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:29.462133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9d4e0014f8d40899201f028d5e88042)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:29.466716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:29.554220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9d4e0014f8d40899201f028d5e88042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.578053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9d4e0014f8d40899201f028d5e88042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.596985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9d4e0014f8d40899201f028d5e88042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.597668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9d4e0014f8d40899201f028d5e88042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.598291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-60fff89a-68e4-4fd7-b461-47edd118573e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9d4e0014f8d40899201f028d5e88042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:29.624560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 821/1666] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:22:29 2022] DELETE /identity/v3/users/c9d4e0014f8d40899201f028d5e88042 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:30.754502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:30.755097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:30.755344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:30.799019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:30.823200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] MFA Rules not processed for user `d945d512ae034ec08838d0a3afde2429`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:30.825188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:30.978437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 846/1667] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:30 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:22:30.989524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:30.990069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:30.990069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:31.030202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] MFA Rules not processed for user `d945d512ae034ec08838d0a3afde2429`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:31.031475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:31.065097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 822/1668] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:30 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:31.085552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:31.086921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:31.251593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c021a8-ecc6-4dfd-b987-e07ca6d525f8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:31.267942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:31.268821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:31.270996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:31.272291 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:31.290357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:31.296663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:31.316557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 847/1669] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:31 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:31.414666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:31.416125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:31.537547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36db815-ab67-4ed7-b23e-3f73835470ab None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:31.548899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:31.553622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:31.553947 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:31.554208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] PATH_INFO: `/v3/projects/458b9db3913441959977889262b2236c` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:31.557889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] RBAC: Authorizing `identity:delete_project(project_id=458b9db3913441959977889262b2236c)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:31.558942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: WARNING py.warnings [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:22:31.558942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: warnings.warn(msg) Aug 27 02:22:31.558942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: Aug 27 02:22:31.566007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:31.672737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '458b9db3913441959977889262b2236c', 'request_id': 'req-cdadf285-4b77-430c-af51-3d3787fcd890'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:31.675521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:31.693053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '458b9db3913441959977889262b2236c', 'request_id': 'req-cdadf285-4b77-430c-af51-3d3787fcd890'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:31.723736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '458b9db3913441959977889262b2236c', 'request_id': 'req-cdadf285-4b77-430c-af51-3d3787fcd890'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:31.736370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:31.738178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:31.766773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] The token cache is being invalidate because project 458b9db3913441959977889262b2236c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:22:31.766978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eadd9ed-7431-42e9-8451-409e12068870 service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:31.769013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:31.770117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:31.790856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 823/1670] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:22:31 2022] DELETE /identity/v3/projects/458b9db3913441959977889262b2236c => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:31.793628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:31.808088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:31.947563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:31.948257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:31.948540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:31.948815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] PATH_INFO: `/v3/projects/f955963f181a4e8f9274cdc63dca5880` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:31.952927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f955963f181a4e8f9274cdc63dca5880)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:31.955760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:32.020250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdadf285-4b77-430c-af51-3d3787fcd890 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:32.030041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:32.030858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.031223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.031710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] PATH_INFO: `/v3/users/771ad70b7a41447fae5d1059d18d7466` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.033358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=771ad70b7a41447fae5d1059d18d7466)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:32.036067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:32.039019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:32.132461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f955963f181a4e8f9274cdc63dca5880', 'request_id': 'req-41ec86a5-582c-4fe5-a07a-8fad04637c45'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.145385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '771ad70b7a41447fae5d1059d18d7466'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.156787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f955963f181a4e8f9274cdc63dca5880', 'request_id': 'req-41ec86a5-582c-4fe5-a07a-8fad04637c45'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.174021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f955963f181a4e8f9274cdc63dca5880', 'request_id': 'req-41ec86a5-582c-4fe5-a07a-8fad04637c45'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.184119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '771ad70b7a41447fae5d1059d18d7466'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.201505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '771ad70b7a41447fae5d1059d18d7466'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.202898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '771ad70b7a41447fae5d1059d18d7466'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.203747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ff85f7a0-0a10-4e55-b15d-6fe2afcdef91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '771ad70b7a41447fae5d1059d18d7466'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.204861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] The token cache is being invalidate because project f955963f181a4e8f9274cdc63dca5880 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:22:32.206429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.208729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-41ec86a5-582c-4fe5-a07a-8fad04637c45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:32.216210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 848/1671] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:22:31 2022] DELETE /identity/v3/projects/f955963f181a4e8f9274cdc63dca5880 => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:32.218906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5cb16e-6603-4785-8a71-7f69b70bccf8 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.221236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5cb16e-6603-4785-8a71-7f69b70bccf8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.223660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5cb16e-6603-4785-8a71-7f69b70bccf8 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.227714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 849/1672] 10.222.0.52 () {54 vars in 912 bytes} [Sat Aug 27 02:22:32 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:22:32.229144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a735a-9151-4b03-bd48-0cca55de11d9 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.231689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a735a-9151-4b03-bd48-0cca55de11d9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.231988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a735a-9151-4b03-bd48-0cca55de11d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.249614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 824/1673] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:22:31 2022] DELETE /identity/v3/users/771ad70b7a41447fae5d1059d18d7466 => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:32.251335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4543f0d1-8f52-46cc-8c65-f1506409a712 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.251889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4543f0d1-8f52-46cc-8c65-f1506409a712 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.252358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4543f0d1-8f52-46cc-8c65-f1506409a712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.278204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-454a735a-9151-4b03-bd48-0cca55de11d9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:32.288215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-454a735a-9151-4b03-bd48-0cca55de11d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:32.356423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4543f0d1-8f52-46cc-8c65-f1506409a712 None None] MFA Rules not processed for user `56aaa029a9c542e0bc4658c5cd55c2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:32.362003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4543f0d1-8f52-46cc-8c65-f1506409a712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:32.454640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 850/1674] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:32.462546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:32.466022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.466313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.466587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.486642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 825/1675] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:22:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:32.517475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:32.518730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:32.544937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 851/1676] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:32.546239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4543f0d1-8f52-46cc-8c65-f1506409a712 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:32.564824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4543f0d1-8f52-46cc-8c65-f1506409a712 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:32.576732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:32.587461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.589941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.590696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.592119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:32.595457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:32.604594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:32.645238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 826/1677] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:22:32.649319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:32.649988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:32.651022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:32.662732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6009daab-f02e-426e-8099-872a840755e2 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:32.671608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:32.672813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.673284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.673913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.678484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:32.680715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:32.683198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bb66fc-db7b-4858-9565-73c4688d5b3d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:32.691784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:32.692475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.694201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.694557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.731233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:32.738518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:32.758348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 827/1678] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:32 2022] POST /identity/v3/projects => generated 363 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:32.762920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:32.766909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 852/1679] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:32.880923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835cfaca-073d-47e8-8738-cec17b449c4c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:32.905674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:32.906556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:32.906836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:32.907107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:32.907920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:32.939643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:32.999633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 828/1680] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:32 2022] POST /identity/v3/users => generated 282 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:33.003240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:33.028024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698245-5d58-436c-9d66-e1faff52aec4 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:33.042482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:33.043116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:33.043393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:33.043851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:33.044768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:33.046635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:33.092742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 853/1681] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:33 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:33.094354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:33.105587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c5da25-3e7e-4c18-ab0d-1f64e682ee97 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:33.112095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7215b29-cf1d-4cf2-bb8e-0097ecb604a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:33.113103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7215b29-cf1d-4cf2-bb8e-0097ecb604a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:33.113479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7215b29-cf1d-4cf2-bb8e-0097ecb604a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:33.114178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7215b29-cf1d-4cf2-bb8e-0097ecb604a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:33.114909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7215b29-cf1d-4cf2-bb8e-0097ecb604a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:33.116699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7215b29-cf1d-4cf2-bb8e-0097ecb604a7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:33.139255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:33.142306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 829/1682] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:33 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:33.152841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e0a4c2-3d51-4feb-9f3d-c651f7ca4a5e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:33.188664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128cbae4-254a-4abe-ba48-20f05d94c652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:33.193635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128cbae4-254a-4abe-ba48-20f05d94c652 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:33.194916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128cbae4-254a-4abe-ba48-20f05d94c652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:33.196734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128cbae4-254a-4abe-ba48-20f05d94c652 admin admin] PATH_INFO: `/v3/projects/bda2e46b3e6841199430c73e8e13f726/users/b61c88120a4a46319370de92564b03b2/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:33.241825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128cbae4-254a-4abe-ba48-20f05d94c652 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bda2e46b3e6841199430c73e8e13f726, user_id=b61c88120a4a46319370de92564b03b2, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:33.246364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128cbae4-254a-4abe-ba48-20f05d94c652 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:33.281491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:33.281954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:33.282357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:33.286000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 854/1683] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:33 2022] PUT /identity/v3/projects/bda2e46b3e6841199430c73e8e13f726/users/b61c88120a4a46319370de92564b03b2/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:22:33.299382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340fe307-c78b-45b6-bd87-7af2c81d9d70 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:33.303687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340fe307-c78b-45b6-bd87-7af2c81d9d70 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:33.304014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340fe307-c78b-45b6-bd87-7af2c81d9d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:33.365091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] MFA Rules not processed for user `26a86351583d49d090b3db09f127d49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:33.366423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:33.376081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-340fe307-c78b-45b6-bd87-7af2c81d9d70 None None] MFA Rules not processed for user `b61c88120a4a46319370de92564b03b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:33.377320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-340fe307-c78b-45b6-bd87-7af2c81d9d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:33.503080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 830/1684] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:22:33.519251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:33.522355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:33.533985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:33.560817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 855/1685] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:33.566198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340fe307-c78b-45b6-bd87-7af2c81d9d70 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:33.592584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:33.714578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe22242-09ee-488b-9b5f-1d20a81fa83c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:33.722160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340fe307-c78b-45b6-bd87-7af2c81d9d70 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:33.732005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:33.734703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:33.735869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:33.736189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:33.736459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:33.737636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:33.738131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:33.738632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:33.740219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:33.745955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:33.756774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:33.758593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:33.771476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 831/1686] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:33.835446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 856/1687] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:33 2022] POST /identity/v3/projects => generated 365 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:33.840605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:33.866088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47136a7e-a8c2-4b34-badf-6bf63233abbb service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:34.065595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:34.078652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:34.094901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e8b1b-22c6-4e03-812f-bcbf899a4d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:34.099572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e8b1b-22c6-4e03-812f-bcbf899a4d5d admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.099917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e8b1b-22c6-4e03-812f-bcbf899a4d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.100539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e8b1b-22c6-4e03-812f-bcbf899a4d5d admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.101552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e8b1b-22c6-4e03-812f-bcbf899a4d5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:34.114827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e8b1b-22c6-4e03-812f-bcbf899a4d5d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:34.172742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 832/1688] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:33 2022] POST /identity/v3/users => generated 289 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:34.173460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e457cf-ca01-4c0a-b03b-e47641870635 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.176149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e457cf-ca01-4c0a-b03b-e47641870635 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.176574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e457cf-ca01-4c0a-b03b-e47641870635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.238727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-64e457cf-ca01-4c0a-b03b-e47641870635 None None] MFA Rules not processed for user `d945d512ae034ec08838d0a3afde2429`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:34.239995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-64e457cf-ca01-4c0a-b03b-e47641870635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:34.284964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882f2cde-e2a2-41ad-b881-0ec1c9c4ee38 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:34.292710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:34.306778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.307063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.307341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.316304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:34.336352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:34.338361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:34.358215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 857/1689] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:34.359453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:34.387234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31314e5c-a30c-46e6-8198-3bc20878c19c service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:34.394869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:34.395613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.396136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.396371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.398363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:34.403358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:34.449321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 858/1690] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:34 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:34.451925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:34.455002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:34.462266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 833/1691] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:34 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:34.463719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8d2277-a961-4ce7-ae95-0654a995b554 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.464005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8d2277-a961-4ce7-ae95-0654a995b554 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.464277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8d2277-a961-4ce7-ae95-0654a995b554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.480931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd00093-af3a-4219-a3a2-d1f77260f5d3 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:34.488086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:34.489205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.489784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.490283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] PATH_INFO: `/v3/projects/c169fcfeeaf245fc81ba9ecbbc73363a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.492190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c169fcfeeaf245fc81ba9ecbbc73363a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:34.495815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:34.566782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-de8d2277-a961-4ce7-ae95-0654a995b554 None None] MFA Rules not processed for user `55640f750d1b49be9c3f79501a63c1ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:34.568413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-de8d2277-a961-4ce7-ae95-0654a995b554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:34.592250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:34.596020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c169fcfeeaf245fc81ba9ecbbc73363a', 'request_id': 'req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:34.642925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c169fcfeeaf245fc81ba9ecbbc73363a', 'request_id': 'req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:34.667160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c169fcfeeaf245fc81ba9ecbbc73363a', 'request_id': 'req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:34.667829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:34.705926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] The token cache is being invalidate because project c169fcfeeaf245fc81ba9ecbbc73363a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:22:34.705926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:34.708449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:34.715514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 859/1692] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:22:34 2022] DELETE /identity/v3/projects/c169fcfeeaf245fc81ba9ecbbc73363a => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:34.716407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:34.723043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:34.783369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 834/1693] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:34 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:34.795118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4a921-4de6-41ad-baf5-b1e165d687bb None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.795283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4a921-4de6-41ad-baf5-b1e165d687bb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.795557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4a921-4de6-41ad-baf5-b1e165d687bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.877356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2cc4a921-4de6-41ad-baf5-b1e165d687bb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:34.879778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2cc4a921-4de6-41ad-baf5-b1e165d687bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:34.911308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8abf0e0-d423-4bcb-b4ce-a7bfd50d88a7 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:34.921480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 835/1694] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:34.924305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50dbde-8053-45bb-9c63-6d2ca266e385 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.925631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50dbde-8053-45bb-9c63-6d2ca266e385 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.929103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50dbde-8053-45bb-9c63-6d2ca266e385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.933962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a323b-7cae-4374-8c55-6dc2f389871e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:34.934662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a323b-7cae-4374-8c55-6dc2f389871e admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:34.934980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a323b-7cae-4374-8c55-6dc2f389871e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:34.935218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a323b-7cae-4374-8c55-6dc2f389871e admin admin] PATH_INFO: `/v3/projects/ee1223564d874629bd9e195f3afbb7e3/users/efb16ab9dddc4ae7be9191882acc69f3/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:34.969736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a323b-7cae-4374-8c55-6dc2f389871e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee1223564d874629bd9e195f3afbb7e3, user_id=efb16ab9dddc4ae7be9191882acc69f3, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:34.976207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a323b-7cae-4374-8c55-6dc2f389871e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:34.995379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-bd50dbde-8053-45bb-9c63-6d2ca266e385 None None] MFA Rules not processed for user `55640f750d1b49be9c3f79501a63c1ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:34.996852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bd50dbde-8053-45bb-9c63-6d2ca266e385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:35.009730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 860/1695] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:34 2022] PUT /identity/v3/projects/ee1223564d874629bd9e195f3afbb7e3/users/efb16ab9dddc4ae7be9191882acc69f3/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:35.015618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:35.017175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:35.018230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:35.081216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:35.082479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:35.155225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 836/1696] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:34 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:35.158099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50dbde-8053-45bb-9c63-6d2ca266e385 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:35.298517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 861/1697] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:35.303355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:35.350132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:35.378592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50dbde-8053-45bb-9c63-6d2ca266e385 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:35.396991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:35.398289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:35.398649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:35.399087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:35.399884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:35.402396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:35.435013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 837/1698] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:35 2022] GET /identity/v3/roles => generated 1420 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:35.436886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:35.452461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bec728-4ab9-4d2d-9e68-4f241cb46f84 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:35.467357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:35.469593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:35.470181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:35.471393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:35.472351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:35.474269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:35.477132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:35.513809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 838/1699] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:35.515774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:35.541997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:35.544361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f25bc8a-3753-436c-b02b-b45f0bc075ee admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:35.563404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:35.564090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:35.564368 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:35.564637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] PATH_INFO: `/v3/projects/ee1223564d874629bd9e195f3afbb7e3/users/efb16ab9dddc4ae7be9191882acc69f3/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:35.573600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee1223564d874629bd9e195f3afbb7e3, user_id=efb16ab9dddc4ae7be9191882acc69f3, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:35.579999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:35.586514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bceb35-ec48-4b09-9c77-4cf5eb7e09c4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:35.601598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4245f0-e71f-437d-b4ba-84f9866d6077 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:35.602255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4245f0-e71f-437d-b4ba-84f9866d6077 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:35.604681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4245f0-e71f-437d-b4ba-84f9866d6077 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:35.604999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4245f0-e71f-437d-b4ba-84f9866d6077 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:35.611854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 839/1700] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:35 2022] PUT /identity/v3/projects/ee1223564d874629bd9e195f3afbb7e3/users/efb16ab9dddc4ae7be9191882acc69f3/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:35.623050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:35.641902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4245f0-e71f-437d-b4ba-84f9866d6077 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:35.643656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4245f0-e71f-437d-b4ba-84f9866d6077 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:35.669230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 862/1701] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:35 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:35.673569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:35.676059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:35.676059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:35.737668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:35.751965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] MFA Rules not processed for user `efb16ab9dddc4ae7be9191882acc69f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:35.756925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:35.787174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1d5e6-7a0d-4e27-a183-81a456c6c2fc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:35.803777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:35.805381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:35.806077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:35.807159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:35.816061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:35.829712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:35.941448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 840/1702] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:35 2022] POST /identity/v3/projects => generated 365 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:35.946906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:35.968017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 863/1703] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:35 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:35.969039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:35.974128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:36.220021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5662f0-1eb6-4e3d-ba59-6915739df509 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:36.235158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd929b63-8202-4bc4-8fce-68f8859e06f2 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:36.272357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:36.275373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.275691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.275970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.276649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:36.286874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:36.348978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 841/1704] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:35 2022] POST /identity/v3/users => generated 283 bytes in 403 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:36.350651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:36.401904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b03165a-1f6f-4492-9a71-d3722ef86356 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:36.423543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:36.436579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.437637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.437838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.448001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:36.452678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.453310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.453310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.454021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:36.472436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:36.544940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 842/1705] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:36 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:36.545872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:36.551341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:36.593455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1377d527-23f3-483d-aba0-4555aef0cb90 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:36.605641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54a462b-8a07-41ef-89ce-44ba4c1029d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:36.605641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54a462b-8a07-41ef-89ce-44ba4c1029d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.605641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54a462b-8a07-41ef-89ce-44ba4c1029d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.605641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54a462b-8a07-41ef-89ce-44ba4c1029d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.614504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:36.614987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b54a462b-8a07-41ef-89ce-44ba4c1029d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:36.709992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:36.711361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:36.767961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 864/1706] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:35 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 799 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:36.793302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:36.847763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676e7f3a-2771-41ab-9d9f-958132e203a6 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:36.872035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d01e2a-6445-4f49-8039-8e387b424647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:36.872677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d01e2a-6445-4f49-8039-8e387b424647 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.872957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d01e2a-6445-4f49-8039-8e387b424647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.876459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d01e2a-6445-4f49-8039-8e387b424647 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.877317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d01e2a-6445-4f49-8039-8e387b424647 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:36.885720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d01e2a-6445-4f49-8039-8e387b424647 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:36.910009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 865/1707] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:36 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:36.911879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d475f6f-dd97-4ef9-93a7-8232b02acdcb None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.912166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d475f6f-dd97-4ef9-93a7-8232b02acdcb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.913355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d475f6f-dd97-4ef9-93a7-8232b02acdcb None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.915694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 866/1708] 10.222.0.52 () {54 vars in 912 bytes} [Sat Aug 27 02:22:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:22:36.916537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d475f6f-dd97-4ef9-93a7-8232b02acdcb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:36.928479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54a462b-8a07-41ef-89ce-44ba4c1029d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:36.930068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54a462b-8a07-41ef-89ce-44ba4c1029d1 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:36.941487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d475f6f-dd97-4ef9-93a7-8232b02acdcb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:36.950406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:36.951830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.952092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.952416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] PATH_INFO: `/v3/projects/7f548d346c764f94817872e8e433cd1d/users/52e4e19d310d400bac7d54bdc88ab6a9/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.955310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 843/1709] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:36 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:36.966660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e3e90-6743-4829-8e5d-12df5bc70222 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:36.966943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e3e90-6743-4829-8e5d-12df5bc70222 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:36.967217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e3e90-6743-4829-8e5d-12df5bc70222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:36.983859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f548d346c764f94817872e8e433cd1d, user_id=52e4e19d310d400bac7d54bdc88ab6a9, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:36.988301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:37.015040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 867/1710] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:36 2022] PUT /identity/v3/projects/7f548d346c764f94817872e8e433cd1d/users/52e4e19d310d400bac7d54bdc88ab6a9/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:37.019111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:37.052587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7f8e3e90-6743-4829-8e5d-12df5bc70222 None None] MFA Rules not processed for user `56aaa029a9c542e0bc4658c5cd55c2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:37.054266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7f8e3e90-6743-4829-8e5d-12df5bc70222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:37.203360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ffbb6-3859-44c6-bf02-72e18f7d7243 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:37.218319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:37.219329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:37.221590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:37.222536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:37.222697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:37.228380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:37.278395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 844/1711] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:22:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:37.279882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedfb198-ac92-4a50-b442-dbf71dbb4b1a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:37.280069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedfb198-ac92-4a50-b442-dbf71dbb4b1a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:37.280360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedfb198-ac92-4a50-b442-dbf71dbb4b1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:37.361512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-dedfb198-ac92-4a50-b442-dbf71dbb4b1a None None] MFA Rules not processed for user `52e4e19d310d400bac7d54bdc88ab6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:37.362770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dedfb198-ac92-4a50-b442-dbf71dbb4b1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:37.440315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:37.443124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:37.491401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 868/1712] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:37 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:37.492533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:37.498779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:37.540127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d518529-5e9c-44f3-b68b-61b2fdaf657a service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:37.561092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:37.561768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:37.562048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:37.562320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:37.574581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:37.576254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:37.601369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 869/1713] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:37.627911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 845/1714] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:37 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:37.633002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:37.744115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:37.815719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d4176b-3527-4d39-9fec-be9ce548a03c service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:37.839562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:37.842583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:37.843704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:37.845822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:37.851897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:37.860238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:37.958236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 870/1715] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:37 2022] POST /identity/v3/projects => generated 365 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:37.962318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedfb198-ac92-4a50-b442-dbf71dbb4b1a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:37.977992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedfb198-ac92-4a50-b442-dbf71dbb4b1a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:38.048276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:38.048970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.049449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.049922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.051495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:38.054733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:38.083380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 846/1716] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:37 2022] POST /identity/v3/users => generated 289 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:22:38.087712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:38.104603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf7894f-9c00-44ea-8e18-9af0e30b1175 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:38.115041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7db4a2b-26a7-435b-92ad-b2a14b47810c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:38.116026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7db4a2b-26a7-435b-92ad-b2a14b47810c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.116369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7db4a2b-26a7-435b-92ad-b2a14b47810c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.116798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7db4a2b-26a7-435b-92ad-b2a14b47810c admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.117703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7db4a2b-26a7-435b-92ad-b2a14b47810c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:38.119658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7db4a2b-26a7-435b-92ad-b2a14b47810c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:38.134773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 871/1717] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:38 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:38.138871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:38.165069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec9e268-e863-42b8-abe0-a38cb4dedb0e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.166772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8814ac-1f2e-4c45-b483-773a07226771 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:38.167694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec9e268-e863-42b8-abe0-a38cb4dedb0e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.167974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec9e268-e863-42b8-abe0-a38cb4dedb0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.181598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:38.182287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.182559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.182831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] PATH_INFO: `/v3/projects/acc6091cb47243c291482cc9fdcdeaca/users/bbc4a9ad392c43bb8e1e57e5fb6cd3ad/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.238645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-bec9e268-e863-42b8-abe0-a38cb4dedb0e None None] MFA Rules not processed for user `26a86351583d49d090b3db09f127d49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:38.241949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bec9e268-e863-42b8-abe0-a38cb4dedb0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:38.252467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=acc6091cb47243c291482cc9fdcdeaca, user_id=bbc4a9ad392c43bb8e1e57e5fb6cd3ad, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:38.256419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:38.279538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 847/1718] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:38 2022] PUT /identity/v3/projects/acc6091cb47243c291482cc9fdcdeaca/users/bbc4a9ad392c43bb8e1e57e5fb6cd3ad/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:22:38.281429 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:38.341988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 872/1719] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:38.356950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:38.385661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d98b56e-7cca-4ff2-a7e6-6333535a78c4 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:38.397456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c254b8-4b77-46e8-b2a6-48085cf8d319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:38.398132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c254b8-4b77-46e8-b2a6-48085cf8d319 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.398411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c254b8-4b77-46e8-b2a6-48085cf8d319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.398678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c254b8-4b77-46e8-b2a6-48085cf8d319 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.399396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c254b8-4b77-46e8-b2a6-48085cf8d319 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:38.406562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c254b8-4b77-46e8-b2a6-48085cf8d319 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:38.427717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 848/1720] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:38 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:38.432110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec9e268-e863-42b8-abe0-a38cb4dedb0e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:38.466961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec9e268-e863-42b8-abe0-a38cb4dedb0e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:38.490812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:38.493739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.494030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.494308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] PATH_INFO: `/v3/projects/acc6091cb47243c291482cc9fdcdeaca/users/bbc4a9ad392c43bb8e1e57e5fb6cd3ad/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.503150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] RBAC: Authorizing `identity:create_grant(project_id=acc6091cb47243c291482cc9fdcdeaca, user_id=bbc4a9ad392c43bb8e1e57e5fb6cd3ad, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:38.506394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:38.527980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 873/1721] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:38 2022] PUT /identity/v3/projects/acc6091cb47243c291482cc9fdcdeaca/users/bbc4a9ad392c43bb8e1e57e5fb6cd3ad/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:38.539845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0dc365-8b33-441e-8269-0cca173298d3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.539845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0dc365-8b33-441e-8269-0cca173298d3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.540087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0dc365-8b33-441e-8269-0cca173298d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.582576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9b0dc365-8b33-441e-8269-0cca173298d3 None None] MFA Rules not processed for user `bbc4a9ad392c43bb8e1e57e5fb6cd3ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:38.584566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9b0dc365-8b33-441e-8269-0cca173298d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:38.668451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 849/1722] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:38 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:38.684955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:38.804649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd639e8-dbbf-4b02-88fd-5d7ba7388156 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:38.810261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6289f29c-b645-4e10-b377-9463f76785bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:38.811073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289f29c-b645-4e10-b377-9463f76785bc service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:38.813132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289f29c-b645-4e10-b377-9463f76785bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:38.814322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289f29c-b645-4e10-b377-9463f76785bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:38.829290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6289f29c-b645-4e10-b377-9463f76785bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:38.831822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6289f29c-b645-4e10-b377-9463f76785bc service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:38.860173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 874/1723] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:38 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:39.670950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0dc365-8b33-441e-8269-0cca173298d3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:39.775202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0dc365-8b33-441e-8269-0cca173298d3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:39.790089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:39.791498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:39.791833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:39.792042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] PATH_INFO: `/v3/users/52e4e19d310d400bac7d54bdc88ab6a9` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:39.800481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52e4e19d310d400bac7d54bdc88ab6a9)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:39.807450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:39.933162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52e4e19d310d400bac7d54bdc88ab6a9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:39.950414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52e4e19d310d400bac7d54bdc88ab6a9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:39.968304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52e4e19d310d400bac7d54bdc88ab6a9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:39.968940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52e4e19d310d400bac7d54bdc88ab6a9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:39.971182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52e4e19d310d400bac7d54bdc88ab6a9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:40.018856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:40.078456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 850/1724] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:22:39 2022] DELETE /identity/v3/users/52e4e19d310d400bac7d54bdc88ab6a9 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:40.084791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:40.085420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:40.085420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:40.115075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:40.116700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:40.240041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:40.241000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 875/1725] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:40.250984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:40.252789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:40.423896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5021362a-0660-4f53-b06e-09c8a468e6f8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:40.436084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:40.436929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:40.437235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:40.437501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:40.473836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:40.484410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:40.536830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 851/1726] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:22:41.216484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:41.218046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:41.242613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f8b9b-8ad8-4f5d-abd4-7d96d2e00acd None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:41.248421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:41.249340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:41.249749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:41.250346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] PATH_INFO: `/v3/projects/7f548d346c764f94817872e8e433cd1d` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:41.251960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f548d346c764f94817872e8e433cd1d)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:41.254563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:41.316189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f548d346c764f94817872e8e433cd1d', 'request_id': 'req-7c4b9422-20e2-4b77-b44d-691b989fe7c4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.324139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f548d346c764f94817872e8e433cd1d', 'request_id': 'req-7c4b9422-20e2-4b77-b44d-691b989fe7c4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.332753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f548d346c764f94817872e8e433cd1d', 'request_id': 'req-7c4b9422-20e2-4b77-b44d-691b989fe7c4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.358380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] The token cache is being invalidate because project 7f548d346c764f94817872e8e433cd1d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:22:41.358380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.358380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.358380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 876/1727] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:22:41 2022] DELETE /identity/v3/projects/7f548d346c764f94817872e8e433cd1d => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:41.362015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:41.380773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752c584a-c125-40ac-9a89-96f761d72368 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:41.462588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:41.463255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:41.463576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:41.463784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] PATH_INFO: `/v3/users/bbc4a9ad392c43bb8e1e57e5fb6cd3ad` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:41.465285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbc4a9ad392c43bb8e1e57e5fb6cd3ad)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:41.467406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:41.525802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbc4a9ad392c43bb8e1e57e5fb6cd3ad'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.534666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbc4a9ad392c43bb8e1e57e5fb6cd3ad'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.552139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbc4a9ad392c43bb8e1e57e5fb6cd3ad'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.553801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbc4a9ad392c43bb8e1e57e5fb6cd3ad'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.554509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-89eb5181-8499-4ca3-8a4d-45a61f2cceab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbc4a9ad392c43bb8e1e57e5fb6cd3ad'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:41.581281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 852/1728] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:22:41 2022] DELETE /identity/v3/users/bbc4a9ad392c43bb8e1e57e5fb6cd3ad => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:41.997210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:42.001509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:42.023164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4b9422-20e2-4b77-b44d-691b989fe7c4 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:42.135950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:42.135950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.135950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.135950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] PATH_INFO: `/v3/projects/acc6091cb47243c291482cc9fdcdeaca` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.136673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=acc6091cb47243c291482cc9fdcdeaca)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:42.138543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:42.211236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'acc6091cb47243c291482cc9fdcdeaca', 'request_id': 'req-5b96e47e-d138-4e88-afd3-badc67c6ca43'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:42.218121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'acc6091cb47243c291482cc9fdcdeaca', 'request_id': 'req-5b96e47e-d138-4e88-afd3-badc67c6ca43'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:42.227142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'acc6091cb47243c291482cc9fdcdeaca', 'request_id': 'req-5b96e47e-d138-4e88-afd3-badc67c6ca43'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:42.243793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] The token cache is being invalidate because project acc6091cb47243c291482cc9fdcdeaca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:22:42.244081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:42.244812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b96e47e-d138-4e88-afd3-badc67c6ca43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:42.246478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 877/1729] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:22:41 2022] DELETE /identity/v3/projects/acc6091cb47243c291482cc9fdcdeaca => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:42.251671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c986cf-bf38-440c-950b-5701ef9efe74 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.252023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c986cf-bf38-440c-950b-5701ef9efe74 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.252339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c986cf-bf38-440c-950b-5701ef9efe74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.275103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-15c986cf-bf38-440c-950b-5701ef9efe74 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:42.276336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-15c986cf-bf38-440c-950b-5701ef9efe74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:42.281989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:42.360399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 853/1730] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:42.372164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b0aa30-a533-4650-b635-e83f40444e3d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.373392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b0aa30-a533-4650-b635-e83f40444e3d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.373830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b0aa30-a533-4650-b635-e83f40444e3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.385592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:42.400309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-90b0aa30-a533-4650-b635-e83f40444e3d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:42.403115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-90b0aa30-a533-4650-b635-e83f40444e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:42.416386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 878/1731] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:42.420497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c986cf-bf38-440c-950b-5701ef9efe74 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:42.431521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c986cf-bf38-440c-950b-5701ef9efe74 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:42.436285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:42.437086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.437514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.438022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.439671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:42.440218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:42.442588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:42.457582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 854/1732] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:22:42.463038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b0aa30-a533-4650-b635-e83f40444e3d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:42.481739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b0aa30-a533-4650-b635-e83f40444e3d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:42.490242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:42.491365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.491722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.492064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.492800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:42.495254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:42.528976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 879/1733] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:42 2022] POST /identity/v3/projects => generated 357 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:42.533721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:42.544955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982f33ba-d6ba-4ab2-bf3c-8064b24e7427 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:42.619100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:42.619727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.620000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.620266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.620918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:42.622909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:42.659417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 855/1734] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:42 2022] POST /identity/v3/users => generated 279 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:42.665714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:42.672984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d841d1-2e81-4f5c-960b-b6bdb8e30387 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:42.678467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:42.679099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.679364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.679635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.680276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:42.681967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:42.694732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 880/1735] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:42 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:42.699825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:42.712832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6cd6ea-3783-4dcb-9fde-87177c4bac6d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:42.730616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12704dca-7f1c-436b-a8ad-9077b5f85d28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:42.731258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12704dca-7f1c-436b-a8ad-9077b5f85d28 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.731533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12704dca-7f1c-436b-a8ad-9077b5f85d28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.731796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12704dca-7f1c-436b-a8ad-9077b5f85d28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.732486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12704dca-7f1c-436b-a8ad-9077b5f85d28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:42.734235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12704dca-7f1c-436b-a8ad-9077b5f85d28 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:42.755461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 856/1736] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:42 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:42.766194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:42.782300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ea7ba-0bfb-4a46-87c2-794729ef216f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:42.795580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:42.796571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.798182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.798651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] PATH_INFO: `/v3/projects/2405358a6fa84b4e99afe6864eed5eaf/users/9dbdca0071824603af2a1521eb2ea160/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.831941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2405358a6fa84b4e99afe6864eed5eaf, user_id=9dbdca0071824603af2a1521eb2ea160, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:42.834855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:42.853201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 881/1737] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:42 2022] PUT /identity/v3/projects/2405358a6fa84b4e99afe6864eed5eaf/users/9dbdca0071824603af2a1521eb2ea160/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:22:42.866080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c666e-5a63-43ec-81a3-bba29f7e4ec2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:42.866438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c666e-5a63-43ec-81a3-bba29f7e4ec2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:42.866898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c666e-5a63-43ec-81a3-bba29f7e4ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:42.912354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3d6c666e-5a63-43ec-81a3-bba29f7e4ec2 None None] MFA Rules not processed for user `9dbdca0071824603af2a1521eb2ea160`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:42.912354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3d6c666e-5a63-43ec-81a3-bba29f7e4ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:42.989146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 857/1738] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:42 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:42.995519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.095846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9579dd0-69a3-4a2b-a5f1-223a119494b1 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.104774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.105953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.106456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.107107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.108360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:43.113876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:43.157106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 882/1739] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:42 2022] POST /identity/v3/projects => generated 355 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:22:43.161698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c666e-5a63-43ec-81a3-bba29f7e4ec2 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.172739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:43.181080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c666e-5a63-43ec-81a3-bba29f7e4ec2 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.275372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.276151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.276513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.276815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.277865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:43.280027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:43.307915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 858/1740] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:43 2022] POST /identity/v3/users => generated 284 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:43.313457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.324468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596bac03-93d0-4d63-a1cb-169e342e7ba5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.329234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.329845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.330837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.330837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.331022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:43.332592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:43.352053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 883/1741] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:43 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:43.356917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.369670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92bc444-19ec-4470-bd70-3a9fef9d4e24 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.374450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.375156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.375470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.375777 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] PATH_INFO: `/v3/projects/698cec9c23d94e5389cb740a61096b4b/users/ab9be73292e34daa829b42d51d7d3f89/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.394693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=698cec9c23d94e5389cb740a61096b4b, user_id=ab9be73292e34daa829b42d51d7d3f89, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:43.397963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:43.413048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 859/1742] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:43 2022] PUT /identity/v3/projects/698cec9c23d94e5389cb740a61096b4b/users/ab9be73292e34daa829b42d51d7d3f89/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:43.417325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.428897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed3f42e-c247-465e-ae00-fc00886cfbf7 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.505099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42301bf5-1fcc-4cb3-88f6-94c40ca6a834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.505776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42301bf5-1fcc-4cb3-88f6-94c40ca6a834 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.506148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42301bf5-1fcc-4cb3-88f6-94c40ca6a834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.506429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42301bf5-1fcc-4cb3-88f6-94c40ca6a834 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.507176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42301bf5-1fcc-4cb3-88f6-94c40ca6a834 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:43.508914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42301bf5-1fcc-4cb3-88f6-94c40ca6a834 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:43.520861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 884/1743] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:43 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:22:43.526515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.537615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54ceac2-7b4b-452b-9dc8-ecda3040eb8c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.542929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.543710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.544051 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.544390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] PATH_INFO: `/v3/projects/698cec9c23d94e5389cb740a61096b4b/users/ab9be73292e34daa829b42d51d7d3f89/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.546586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=698cec9c23d94e5389cb740a61096b4b, user_id=ab9be73292e34daa829b42d51d7d3f89, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:43.549620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:43.563651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 860/1744] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:43 2022] PUT /identity/v3/projects/698cec9c23d94e5389cb740a61096b4b/users/ab9be73292e34daa829b42d51d7d3f89/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:43.576068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e790036-afa7-46da-b10c-b889e51b9ee5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.577102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e790036-afa7-46da-b10c-b889e51b9ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.577102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e790036-afa7-46da-b10c-b889e51b9ee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.598296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:43.608003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-7e790036-afa7-46da-b10c-b889e51b9ee5 None None] MFA Rules not processed for user `ab9be73292e34daa829b42d51d7d3f89`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:43.609460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7e790036-afa7-46da-b10c-b889e51b9ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:43.699326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 885/1745] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:43 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:43.717887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.720068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:43.799148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07adc483-9908-462f-8980-881b1222f4a1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.802989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3127e2fb-7a0b-4f09-b1e7-cd2e949491f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.803574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3127e2fb-7a0b-4f09-b1e7-cd2e949491f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.803849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3127e2fb-7a0b-4f09-b1e7-cd2e949491f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.804113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3127e2fb-7a0b-4f09-b1e7-cd2e949491f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:43.812863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3127e2fb-7a0b-4f09-b1e7-cd2e949491f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:43.814424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3127e2fb-7a0b-4f09-b1e7-cd2e949491f6 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:43.824984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 861/1746] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:43 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:22:43.917984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e790036-afa7-46da-b10c-b889e51b9ee5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:43.931094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e790036-afa7-46da-b10c-b889e51b9ee5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:43.937187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:43.938390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:43.938985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:43.940231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:44.051618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:44.053336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:44.065803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 886/1747] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:43 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:50.225865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c03889-bb59-4a50-a2c3-36c441816d6a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:50.226668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c03889-bb59-4a50-a2c3-36c441816d6a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:50.226668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c03889-bb59-4a50-a2c3-36c441816d6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:50.295690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c7c03889-bb59-4a50-a2c3-36c441816d6a None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:50.296282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c7c03889-bb59-4a50-a2c3-36c441816d6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:50.355470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:50.424484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 862/1748] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:22:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:50.813007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:50.815265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:50.989191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27723742-2fe0-4d48-8286-1f9dab199088 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:50.995785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:50.996781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:50.997261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:50.997624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:51.023700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:51.026455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:51.044474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 887/1749] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:22:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:22:52.384852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:52.386188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:52.387276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:52.478237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] MFA Rules not processed for user `56aaa029a9c542e0bc4658c5cd55c2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:52.481218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:52.718038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 863/1750] 10.222.0.45 () {58 vars in 1015 bytes} [Sat Aug 27 02:22:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:52.781356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:52.785593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:52.825322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:52.840411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad5ab9e-da91-4a81-8a82-e9c551b100b6 service swift] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:52.856322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338701ae-ef19-454e-93c7-a9981ec1c571 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:52.858641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338701ae-ef19-454e-93c7-a9981ec1c571 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:52.859826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338701ae-ef19-454e-93c7-a9981ec1c571 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:52.866029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338701ae-ef19-454e-93c7-a9981ec1c571 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:52.897917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338701ae-ef19-454e-93c7-a9981ec1c571 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:52.903623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338701ae-ef19-454e-93c7-a9981ec1c571 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:52.956135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 888/1751] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:53.062320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:53.064472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:53.264893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c64444-ab0e-4f60-bdc0-5daa4fa6dfbe None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:53.281798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:53.282472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:53.282745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:53.283375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] PATH_INFO: `/v3/users/7aa09b0963d04b588386f1c0a96e2128` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:53.284843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7aa09b0963d04b588386f1c0a96e2128)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:53.293210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:53.474903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7aa09b0963d04b588386f1c0a96e2128'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:53.514054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7aa09b0963d04b588386f1c0a96e2128'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:53.539374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7aa09b0963d04b588386f1c0a96e2128'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:53.542155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7aa09b0963d04b588386f1c0a96e2128'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:53.543830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7aa09b0963d04b588386f1c0a96e2128'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:53.579794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:53.598593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 864/1752] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:22:53 2022] DELETE /identity/v3/users/7aa09b0963d04b588386f1c0a96e2128 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:53.603814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:53.604327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:53.604748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:53.635073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:53.636892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:53.764091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 889/1753] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:53.791089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:53.793070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:53.934540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe15db8-51d9-47fc-aef5-7147386e67d3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:53.949288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c64118-3f61-4f97-8f3f-9aa030a5f810 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:53.950084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c64118-3f61-4f97-8f3f-9aa030a5f810 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:53.950381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c64118-3f61-4f97-8f3f-9aa030a5f810 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:53.950671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c64118-3f61-4f97-8f3f-9aa030a5f810 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:53.974776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c64118-3f61-4f97-8f3f-9aa030a5f810 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:53.976574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c64118-3f61-4f97-8f3f-9aa030a5f810 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:53.999245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 865/1754] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:54.954678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:54.956905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:54.973282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c5aa0e-0f9c-488b-a33d-e8f287b84da4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:54.979730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:54.980477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:54.980853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:54.981318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] PATH_INFO: `/v3/projects/5c41fc70b9774df08ee9b819edc12ba8` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:54.982556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c41fc70b9774df08ee9b819edc12ba8)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:54.984594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.029643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b5a2d-9962-4760-b80d-57f25106cf31 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.030195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b5a2d-9962-4760-b80d-57f25106cf31 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.030528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b5a2d-9962-4760-b80d-57f25106cf31 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.038336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 866/1755] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:22:55 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:22:55.058670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.059271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.059960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.077419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c41fc70b9774df08ee9b819edc12ba8', 'request_id': 'req-1991a671-d653-4c92-ac55-1ea1fbd5a527'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:55.089799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c41fc70b9774df08ee9b819edc12ba8', 'request_id': 'req-1991a671-d653-4c92-ac55-1ea1fbd5a527'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:55.105320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c41fc70b9774df08ee9b819edc12ba8', 'request_id': 'req-1991a671-d653-4c92-ac55-1ea1fbd5a527'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:55.108519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:55.113732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:55.134831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] The token cache is being invalidate because project 5c41fc70b9774df08ee9b819edc12ba8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:22:55.135262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:55.136121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1991a671-d653-4c92-ac55-1ea1fbd5a527 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:22:55.138215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 890/1756] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:22:54 2022] DELETE /identity/v3/projects/5c41fc70b9774df08ee9b819edc12ba8 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:55.145540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31afa39f-3feb-45a4-916a-5378b4e2dcd5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.145854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31afa39f-3feb-45a4-916a-5378b4e2dcd5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.146253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31afa39f-3feb-45a4-916a-5378b4e2dcd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.184589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-31afa39f-3feb-45a4-916a-5378b4e2dcd5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:55.185676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-31afa39f-3feb-45a4-916a-5378b4e2dcd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:55.314777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 891/1758] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:55.314992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 867/1758] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:22:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:55.317842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da36d3-6e3f-4569-bf60-ac24c295fd4e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.318324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da36d3-6e3f-4569-bf60-ac24c295fd4e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.318817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da36d3-6e3f-4569-bf60-ac24c295fd4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.324210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.326593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:55.346856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fbfb4a-572e-408b-914b-e130a567e8ac None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:55.357256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-85da36d3-6e3f-4569-bf60-ac24c295fd4e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:55.359250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-85da36d3-6e3f-4569-bf60-ac24c295fd4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:55.360632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:55.361701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.362180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.362667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.368577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 892/1759] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:55.372888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85da36d3-6e3f-4569-bf60-ac24c295fd4e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.380081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:55.382706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.383892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85da36d3-6e3f-4569-bf60-ac24c295fd4e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:55.388624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:55.389257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.389539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.389818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.390441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:22:55.390737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:55.392320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.394169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 868/1760] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:22:55.397514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:55.411086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 893/1761] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:22:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:22:55.415487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.426650 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744f1a7d-ab51-41fe-98c3-b00c6c1ca9f2 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:55.431357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:55.431958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.432230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.432495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.433178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:55.435059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.480527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 869/1762] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:55 2022] POST /identity/v3/projects => generated 391 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:22:55.486238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.497425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03da16e7-771c-4c2f-bba1-70e3b04c2c0b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:55.575478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:55.576393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.576827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.577356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.578107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:55.580301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.611558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 894/1763] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:55 2022] POST /identity/v3/users => generated 296 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:55.617381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.633501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c4f3c1-565b-4de3-8bbd-95e9b00efe7f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:55.641417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:55.642557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.643113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.643553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.644709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:55.648373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.673615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 870/1764] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:55 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:22:55.679771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.702481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60198b3-ae2a-491b-8ad7-d3d96fc41889 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:55.709685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54280386-3b90-47a8-8f8b-f969b19338f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:55.710854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54280386-3b90-47a8-8f8b-f969b19338f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.711384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54280386-3b90-47a8-8f8b-f969b19338f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.711959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54280386-3b90-47a8-8f8b-f969b19338f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.713248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54280386-3b90-47a8-8f8b-f969b19338f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:55.716325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54280386-3b90-47a8-8f8b-f969b19338f0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.728347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 895/1765] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:55 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:55.735679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.746744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e3fb2b-9f00-400b-bb44-aa29b0632724 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:55.751649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:55.752301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.752577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.752847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] PATH_INFO: `/v3/projects/ff55582e74a64b959366e03b8aca3526/users/d84378a10abc4681b385fb0cbe6c219a/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.779145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff55582e74a64b959366e03b8aca3526, user_id=d84378a10abc4681b385fb0cbe6c219a, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:55.781821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:55.803298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 871/1766] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:55 2022] PUT /identity/v3/projects/ff55582e74a64b959366e03b8aca3526/users/d84378a10abc4681b385fb0cbe6c219a/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:55.821540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc259a9-e2ad-48d3-a112-dbd4baede472 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:55.822495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc259a9-e2ad-48d3-a112-dbd4baede472 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:55.823915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc259a9-e2ad-48d3-a112-dbd4baede472 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:55.857976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1cc259a9-e2ad-48d3-a112-dbd4baede472 None None] MFA Rules not processed for user `d84378a10abc4681b385fb0cbe6c219a`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:55.859190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1cc259a9-e2ad-48d3-a112-dbd4baede472 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:55.934546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 896/1767] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:55 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:55.940412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:55.978404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:56.022931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e8bbcf-90c6-43a4-a9ed-38f2cb6e11cb admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.028210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.028820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.029094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.029396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.030045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.031967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.072621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 872/1768] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:22:55 2022] POST /identity/v3/projects => generated 391 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:22:56.076961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc259a9-e2ad-48d3-a112-dbd4baede472 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:56.088835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc259a9-e2ad-48d3-a112-dbd4baede472 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.147566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.148218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.148528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.148830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.149549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.151754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.162237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:56.179371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 897/1769] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:22:56 2022] POST /identity/v3/users => generated 302 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:22:56.183606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:56.195624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e31448-a68a-49b0-bf7c-fa94c2892d17 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.200532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.201109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.201416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.201691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.202334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.203986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.217309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 873/1770] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:56 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:22:56.221762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:56.234480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb4c73-4ca8-4963-af10-efbc74616ad4 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.241989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.242980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.243392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.243795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] PATH_INFO: `/v3/projects/0675a000903947b8aab9b161e6a61db8/users/18a2908423f145068379a922ed167d06/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.263005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0675a000903947b8aab9b161e6a61db8, user_id=18a2908423f145068379a922ed167d06, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.265816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.282937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 898/1771] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:56 2022] PUT /identity/v3/projects/0675a000903947b8aab9b161e6a61db8/users/18a2908423f145068379a922ed167d06/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:56.287665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:56.297745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8483bb4-54f8-4c7b-9eeb-70e8ed997385 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.360457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d307ef75-2a68-4b33-a21e-df6471259c21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.361294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d307ef75-2a68-4b33-a21e-df6471259c21 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.361702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d307ef75-2a68-4b33-a21e-df6471259c21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.362118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d307ef75-2a68-4b33-a21e-df6471259c21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.363178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d307ef75-2a68-4b33-a21e-df6471259c21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.364875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d307ef75-2a68-4b33-a21e-df6471259c21 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.381688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 874/1772] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:22:56 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:22:56.383333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:56.397832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e773416-7855-43f6-a556-d82f5fabf232 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.405304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.406435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.406899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.406899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] PATH_INFO: `/v3/projects/0675a000903947b8aab9b161e6a61db8/users/18a2908423f145068379a922ed167d06/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.413888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0675a000903947b8aab9b161e6a61db8, user_id=18a2908423f145068379a922ed167d06, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.413888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.431068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 899/1773] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:22:56 2022] PUT /identity/v3/projects/0675a000903947b8aab9b161e6a61db8/users/18a2908423f145068379a922ed167d06/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:22:56.440893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.441308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.441570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.480876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] MFA Rules not processed for user `18a2908423f145068379a922ed167d06`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:22:56.482142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:56.609424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 875/1774] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:22:56 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:22:56.629360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:56.632566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:56.706412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833a4ca9-e73a-4cf0-b5de-c35ffcc9f972 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.710923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53c7bc4-1ba3-4567-a479-8bbd242571a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.711546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53c7bc4-1ba3-4567-a479-8bbd242571a1 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.711820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53c7bc4-1ba3-4567-a479-8bbd242571a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.712146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53c7bc4-1ba3-4567-a479-8bbd242571a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.724242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53c7bc4-1ba3-4567-a479-8bbd242571a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.726160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53c7bc4-1ba3-4567-a479-8bbd242571a1 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.736749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 900/1775] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:56 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:22:56.803327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:22:56.804804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:22:56.816867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e8ac16-3542-4da8-85da-ffd27c033c6d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:22:56.822070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8508f81d-9f53-466c-985d-da073b2f7a8c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:22:56.823046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8508f81d-9f53-466c-985d-da073b2f7a8c service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:22:56.823499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8508f81d-9f53-466c-985d-da073b2f7a8c service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:22:56.824179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8508f81d-9f53-466c-985d-da073b2f7a8c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:22:56.844155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:22:56.893336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8508f81d-9f53-466c-985d-da073b2f7a8c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:22:56.894996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8508f81d-9f53-466c-985d-da073b2f7a8c service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:22:56.905726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 876/1776] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:22:56 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:06.731721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:INSERT=3,keystone:SELECT=43 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:06.901561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=19 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:16.239744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e2fffd-d5a4-49cf-b327-fdc766cf5dbc None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:16.240714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e2fffd-d5a4-49cf-b327-fdc766cf5dbc None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:16.240714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e2fffd-d5a4-49cf-b327-fdc766cf5dbc None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:16.242596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 901/1777] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:23:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:23:16.262471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed21667-54d7-4bc0-ae5a-d95ed15e731f None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:16.263798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed21667-54d7-4bc0-ae5a-d95ed15e731f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:16.264274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed21667-54d7-4bc0-ae5a-d95ed15e731f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:16.299733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-aed21667-54d7-4bc0-ae5a-d95ed15e731f None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:16.302301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-aed21667-54d7-4bc0-ae5a-d95ed15e731f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:16.516429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e2fffd-d5a4-49cf-b327-fdc766cf5dbc None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:16.521920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 877/1778] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:23:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:16.705432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e2fffd-d5a4-49cf-b327-fdc766cf5dbc None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:16.711471 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:16.712134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:16.712407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:16.712690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] PATH_INFO: `/v3/users/d84378a10abc4681b385fb0cbe6c219a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:16.714235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d84378a10abc4681b385fb0cbe6c219a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:16.715405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: WARNING py.warnings [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 27 02:23:16.715405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: warnings.warn(msg) Aug 27 02:23:16.715405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: Aug 27 02:23:16.716763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:16.875937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd84378a10abc4681b385fb0cbe6c219a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:16.976665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd84378a10abc4681b385fb0cbe6c219a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:16.979212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd84378a10abc4681b385fb0cbe6c219a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.003826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd84378a10abc4681b385fb0cbe6c219a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.020876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd84378a10abc4681b385fb0cbe6c219a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.031225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 902/1779] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:16 2022] DELETE /identity/v3/users/d84378a10abc4681b385fb0cbe6c219a => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:17.039126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:17.039612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:17.039977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:17.094525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:17.097228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:17.100291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:17.102349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:17.254857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 878/1780] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:17.269716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:17.272747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:17.297351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9f717d-14b8-4ba6-a6c3-f35c238b980c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:17.308274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:17.309221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:17.309658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:17.310207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:17.313169 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:17.329319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:17.331155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:17.355882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 903/1781] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:17.420591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf387e3-fd19-43db-bc80-44ca7fa702f1 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:17.425704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:17.426329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:17.426601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:17.427064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:17.439774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:17.441437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:17.455553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 879/1782] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:17.843542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:17.845509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:17.861017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:17.868857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d619d8-c0da-4a03-88f9-a921596b88fc service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:17.881516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:17.882517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:17.883088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:17.883448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] PATH_INFO: `/v3/projects/ff55582e74a64b959366e03b8aca3526` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:17.885742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff55582e74a64b959366e03b8aca3526)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:17.887232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:17.961670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff55582e74a64b959366e03b8aca3526', 'request_id': 'req-f98b9c74-a236-4987-8aed-d18b7e020543'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.970541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff55582e74a64b959366e03b8aca3526', 'request_id': 'req-f98b9c74-a236-4987-8aed-d18b7e020543'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.977793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff55582e74a64b959366e03b8aca3526', 'request_id': 'req-f98b9c74-a236-4987-8aed-d18b7e020543'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.992769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] The token cache is being invalidate because project ff55582e74a64b959366e03b8aca3526 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:17.993055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.993746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:17.995266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 904/1783] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:17 2022] DELETE /identity/v3/projects/ff55582e74a64b959366e03b8aca3526 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:17.999990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:18.016422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee63439-35fb-4aab-b7cb-2fd8f4f86047 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:18.059924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:18.089873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:18.090544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:18.090820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:18.091095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] PATH_INFO: `/v3/users/18a2908423f145068379a922ed167d06` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:18.093063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18a2908423f145068379a922ed167d06)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:18.094621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:18.188336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18a2908423f145068379a922ed167d06'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.203037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18a2908423f145068379a922ed167d06'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.214089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18a2908423f145068379a922ed167d06'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.214811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18a2908423f145068379a922ed167d06'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.215491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18a2908423f145068379a922ed167d06'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.240094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 880/1784] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:17 2022] DELETE /identity/v3/users/18a2908423f145068379a922ed167d06 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:18.595281 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:18.600149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:18.609301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:18.610835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:18.621801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b9c74-a236-4987-8aed-d18b7e020543 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:18.699740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:18.700618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:18.701025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:18.701683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] PATH_INFO: `/v3/users/b61c88120a4a46319370de92564b03b2` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:18.705544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b61c88120a4a46319370de92564b03b2)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:18.707475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:18.754958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a8bcc2-691d-4dc9-a1e5-e561751470b2 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:18.760305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:18.760944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:18.761310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:18.761654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] PATH_INFO: `/v3/projects/0675a000903947b8aab9b161e6a61db8` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:18.762811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0675a000903947b8aab9b161e6a61db8)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:18.764312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b61c88120a4a46319370de92564b03b2'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.764698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:18.784556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b61c88120a4a46319370de92564b03b2'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.786743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b61c88120a4a46319370de92564b03b2'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.796902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b61c88120a4a46319370de92564b03b2'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.806163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-51866c25-ca2a-49b6-bc45-10aded5b6ee1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b61c88120a4a46319370de92564b03b2'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.815947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 905/1785] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:18 2022] DELETE /identity/v3/users/b61c88120a4a46319370de92564b03b2 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:18.817339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:18.817974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:18.818157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:18.832485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:18.840144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0675a000903947b8aab9b161e6a61db8', 'request_id': 'req-5e102420-596b-4b51-86c1-4d6171b4a9e3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.841204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:18.843522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:18.850310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0675a000903947b8aab9b161e6a61db8', 'request_id': 'req-5e102420-596b-4b51-86c1-4d6171b4a9e3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.851945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:18.859906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0675a000903947b8aab9b161e6a61db8', 'request_id': 'req-5e102420-596b-4b51-86c1-4d6171b4a9e3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.874898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] The token cache is being invalidate because project 0675a000903947b8aab9b161e6a61db8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:18.875188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.876042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5e102420-596b-4b51-86c1-4d6171b4a9e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:18.877581 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 881/1786] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:18 2022] DELETE /identity/v3/projects/0675a000903947b8aab9b161e6a61db8 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:18.882770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548aaa45-1b05-4e5f-b798-87a05633d6dc None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:18.883051 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548aaa45-1b05-4e5f-b798-87a05633d6dc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:18.883334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548aaa45-1b05-4e5f-b798-87a05633d6dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:18.908857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-548aaa45-1b05-4e5f-b798-87a05633d6dc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:18.910386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-548aaa45-1b05-4e5f-b798-87a05633d6dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:18.933290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 906/1787] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:18.945951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:18.948386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:18.989843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 882/1788] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:18.996779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2797378-8484-444a-8eae-a8e3d7d30355 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:18.997339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2797378-8484-444a-8eae-a8e3d7d30355 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:18.997768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2797378-8484-444a-8eae-a8e3d7d30355 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.020682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e2797378-8484-444a-8eae-a8e3d7d30355 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:19.022272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e2797378-8484-444a-8eae-a8e3d7d30355 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:19.034030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 883/1789] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:19.035265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2797378-8484-444a-8eae-a8e3d7d30355 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.036546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066742b1-5d4f-4e5a-b1c2-bcd4af4155b5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.043384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.044405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.044871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.045360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.046971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2797378-8484-444a-8eae-a8e3d7d30355 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.053679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.054702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.055029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.055184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.056547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:19.056859 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.058512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.058708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.060556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.077027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 884/1790] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:19.081611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.090108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 907/1791] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:19.095769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b04feed-c6c2-4110-afdf-b83c58ec0db5 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.102020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.102873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.103193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.103651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.104407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.106618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.139167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 885/1792] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:19 2022] POST /identity/v3/projects => generated 379 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:19.159008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.182705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83d081-4372-4e16-9592-e668f2b3398e service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.285157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.285819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.286246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.286411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.287156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.289173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.320374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 908/1793] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:19 2022] POST /identity/v3/users => generated 290 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:19.325017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.339527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca34a07d-7e8e-49b1-a26e-28bbc7015c56 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.354654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.354654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.354879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.355209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.355750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.358363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.375755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 886/1794] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:19 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:19.378242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.389863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0ae819-4deb-4857-9bfe-f1a95ea8d59b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.396093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a721990-1e74-40af-bfa7-74fb2479f121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.397074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a721990-1e74-40af-bfa7-74fb2479f121 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.397574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a721990-1e74-40af-bfa7-74fb2479f121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.398186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a721990-1e74-40af-bfa7-74fb2479f121 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.398916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a721990-1e74-40af-bfa7-74fb2479f121 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.401638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a721990-1e74-40af-bfa7-74fb2479f121 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.418017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 909/1795] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:19 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:19.421113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.434110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303dc846-f60b-4119-9c13-3bf0ba962385 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.441231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.441938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.442218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.442614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] PATH_INFO: `/v3/projects/d2911f74871042e5ac518303c21bde26/users/fe6fd08203e2402585695d3c51e92b7a/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.465597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d2911f74871042e5ac518303c21bde26, user_id=fe6fd08203e2402585695d3c51e92b7a, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.469495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.493332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 887/1796] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:19 2022] PUT /identity/v3/projects/d2911f74871042e5ac518303c21bde26/users/fe6fd08203e2402585695d3c51e92b7a/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:19.509569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e71add-641d-474e-8b1a-2c4ae729427c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.510054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e71add-641d-474e-8b1a-2c4ae729427c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.510779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e71add-641d-474e-8b1a-2c4ae729427c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.543817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.545701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:19.573955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-68e71add-641d-474e-8b1a-2c4ae729427c None None] MFA Rules not processed for user `fe6fd08203e2402585695d3c51e92b7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:19.576873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-68e71add-641d-474e-8b1a-2c4ae729427c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:19.602908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:19.680588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9082d9cc-27d6-4eee-b23a-4ebb838c429a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.692983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.694097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.696311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.696603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] PATH_INFO: `/v3/projects/bda2e46b3e6841199430c73e8e13f726` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.699871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bda2e46b3e6841199430c73e8e13f726)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.701875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.752920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:19.753215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 910/1797] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:19.758361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e71add-641d-474e-8b1a-2c4ae729427c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.782116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e71add-641d-474e-8b1a-2c4ae729427c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.787979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.788607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.788896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.789227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.789850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.792391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.814161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bda2e46b3e6841199430c73e8e13f726', 'request_id': 'req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:19.829564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bda2e46b3e6841199430c73e8e13f726', 'request_id': 'req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:19.831399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 911/1798] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:19 2022] POST /identity/v3/projects => generated 377 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:19.836295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.837020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bda2e46b3e6841199430c73e8e13f726', 'request_id': 'req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:19.848364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] The token cache is being invalidate because project bda2e46b3e6841199430c73e8e13f726 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:19.848775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:19.849716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:19.851566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 888/1799] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:19 2022] DELETE /identity/v3/projects/bda2e46b3e6841199430c73e8e13f726 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:19.855475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.857012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:19.935538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8afb6-b95e-45da-aaa4-264bb8959658 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.941053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.941703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.942028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.942297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.943046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.945045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.953164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e593c3f-7b78-4f6d-a6cd-2afeb1187a72 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:19.967275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:19.968616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:19.968616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:19.968616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] PATH_INFO: `/v3/users/efb16ab9dddc4ae7be9191882acc69f3` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:19.969986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] RBAC: Authorizing `identity:delete_user(user_id=efb16ab9dddc4ae7be9191882acc69f3)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:19.970986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:19.983779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 912/1800] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:19 2022] POST /identity/v3/users => generated 289 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:19.988481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:19.990808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:20.004363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbff644-1b72-49bb-a927-fbbce9461c14 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.013464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.014423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.014959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.015370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.016165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.018032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.032041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 913/1801] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:19 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:20.034684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.042571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efb16ab9dddc4ae7be9191882acc69f3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.048108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec3910f-3ce9-4136-b9b0-bbc330a023df admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.053690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efb16ab9dddc4ae7be9191882acc69f3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.053860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.054294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.054580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.054842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.055644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.057252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.063196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efb16ab9dddc4ae7be9191882acc69f3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.063917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efb16ab9dddc4ae7be9191882acc69f3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.064579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-49749d92-5f28-4aba-85dc-2780e011129a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efb16ab9dddc4ae7be9191882acc69f3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.077777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 914/1802] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:20 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:20.081715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.083786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:20.093745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 889/1803] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:19 2022] DELETE /identity/v3/users/efb16ab9dddc4ae7be9191882acc69f3 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:20.099696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070030b4-021c-4ffc-b968-1a80086e20f2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.172314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.173005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.173334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.173701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] PATH_INFO: `/v3/projects/331d4970745b449bad65671bec6351c8/users/eb5dae33b5fc401caed59579c721736e/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.185152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:20.197843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=331d4970745b449bad65671bec6351c8, user_id=eb5dae33b5fc401caed59579c721736e, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.203258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.219737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 915/1804] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:20 2022] PUT /identity/v3/projects/331d4970745b449bad65671bec6351c8/users/eb5dae33b5fc401caed59579c721736e/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:20.230783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.231172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.231506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.270063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] MFA Rules not processed for user `eb5dae33b5fc401caed59579c721736e`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:20.270063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:20.274001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:20.340608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 890/1805] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:20.345315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.389846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.391484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:20.425412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995341f-e6b4-4b4d-b6b2-41ec115a6e0c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.430369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.430883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.431157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.431520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.432148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.434219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.464306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 916/1806] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:20 2022] POST /identity/v3/projects => generated 377 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:20.468291 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.482980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64be3e4-bc25-40e4-a297-972e5ccd380c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.561398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.562589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.563124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.563655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] PATH_INFO: `/v3/projects/ee1223564d874629bd9e195f3afbb7e3` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.565739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee1223564d874629bd9e195f3afbb7e3)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.570051 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.578288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef60c398-7071-42ba-a330-783dc44edd95 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.585740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.586385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.586751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.587267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.587931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.590269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.617442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 917/1807] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:20 2022] POST /identity/v3/users => generated 295 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:20.621932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.632812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee1223564d874629bd9e195f3afbb7e3', 'request_id': 'req-77f987ec-3dcf-4850-be58-6288bf32194c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.633836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e33438b-6816-4d39-9ac2-4e104866b369 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.641776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee1223564d874629bd9e195f3afbb7e3', 'request_id': 'req-77f987ec-3dcf-4850-be58-6288bf32194c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.642956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:20.649765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee1223564d874629bd9e195f3afbb7e3', 'request_id': 'req-77f987ec-3dcf-4850-be58-6288bf32194c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.663895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] The token cache is being invalidate because project ee1223564d874629bd9e195f3afbb7e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:20.664192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.664883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f987ec-3dcf-4850-be58-6288bf32194c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:20.669829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 891/1808] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:20 2022] DELETE /identity/v3/projects/ee1223564d874629bd9e195f3afbb7e3 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:20.671785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72065c-a600-499e-87ee-bf4adfc60639 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.672206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72065c-a600-499e-87ee-bf4adfc60639 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.672960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72065c-a600-499e-87ee-bf4adfc60639 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.693408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ea72065c-a600-499e-87ee-bf4adfc60639 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:20.694673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ea72065c-a600-499e-87ee-bf4adfc60639 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:20.699514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.700464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.701189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.701549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.702494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.704198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.705594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 892/1809] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:20.707308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:20.715065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aea6e9-b9aa-47b7-ab6f-b5f8d06e25b0 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.715370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aea6e9-b9aa-47b7-ab6f-b5f8d06e25b0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.715606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aea6e9-b9aa-47b7-ab6f-b5f8d06e25b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.718484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 918/1810] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:20 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:20.722997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.724996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:20.748223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-30aea6e9-b9aa-47b7-ab6f-b5f8d06e25b0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:20.748449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39103c4-a728-470a-9b9f-4e7b6792dd74 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.749609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-30aea6e9-b9aa-47b7-ab6f-b5f8d06e25b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:20.754353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.755181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.755488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.755952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] PATH_INFO: `/v3/projects/73184ddaf13f4353b1de04407b614d99/users/8562555379ec44aa8ebfd83ec0a1eb78/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.762352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 893/1811] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:20.771243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30aea6e9-b9aa-47b7-ab6f-b5f8d06e25b0 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.776048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73184ddaf13f4353b1de04407b614d99, user_id=8562555379ec44aa8ebfd83ec0a1eb78, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.779078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.784958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30aea6e9-b9aa-47b7-ab6f-b5f8d06e25b0 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.792177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.792905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.793276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.793652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.794422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:20.794968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.798214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.803588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 919/1812] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:20 2022] PUT /identity/v3/projects/73184ddaf13f4353b1de04407b614d99/users/8562555379ec44aa8ebfd83ec0a1eb78/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:20.804799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.813785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 894/1813] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:20.818151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.838476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0495c3da-1e38-435f-99ad-66ad55bf8fcd admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.918623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8561ae-17af-4eaa-b22d-bd3f704eb6d1 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.923859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.924597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.925913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.926591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.927139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.927467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.928244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.928392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.928701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.929505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.930678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.931317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.942801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 920/1814] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:20 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:20.950851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.962790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127afd15-29f8-4911-9823-8e6768890bb2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:20.968022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16dfadc-8469-4c81-9765-228766e7badc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:20.969110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16dfadc-8469-4c81-9765-228766e7badc admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:20.969247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 895/1815] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:20 2022] POST /identity/v3/projects => generated 367 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:20.969415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16dfadc-8469-4c81-9765-228766e7badc admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:20.969790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16dfadc-8469-4c81-9765-228766e7badc admin admin] PATH_INFO: `/v3/projects/73184ddaf13f4353b1de04407b614d99/users/8562555379ec44aa8ebfd83ec0a1eb78/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:20.972061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16dfadc-8469-4c81-9765-228766e7badc admin admin] RBAC: Authorizing `identity:create_grant(project_id=73184ddaf13f4353b1de04407b614d99, user_id=8562555379ec44aa8ebfd83ec0a1eb78, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:20.975178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16dfadc-8469-4c81-9765-228766e7badc admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:20.975476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:20.991929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 921/1816] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:20 2022] PUT /identity/v3/projects/73184ddaf13f4353b1de04407b614d99/users/8562555379ec44aa8ebfd83ec0a1eb78/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:21.003564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.003987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.004387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.040862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] MFA Rules not processed for user `8562555379ec44aa8ebfd83ec0a1eb78`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:21.042344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:21.072453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e900d-2599-4e15-934d-7114a7280486 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.131047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 922/1817] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:21 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:21.143492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.143742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:21.146256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:21.157847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.159003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.159942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.160355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.161140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.163344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.190976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 896/1818] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:20 2022] POST /identity/v3/users => generated 284 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:21.201290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.218929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:21.219703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254722a6-8d2c-4107-9a0a-8aa5265e34cc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.226388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.227052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.227347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.227636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.229492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.230173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.244695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9138d6-3a07-40df-82e8-da85894b9792 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.245680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 897/1819] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:21 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:21.253986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.258769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.259413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.259693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.259960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.266003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f253ecd3-fb01-486c-a045-1be0d04c033e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.272384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.272837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.273214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.273549 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.273876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.274599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.275522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.276360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.289275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 898/1820] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:21 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:21.293982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 923/1821] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:21 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:23:21.295995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.308984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4e8dbc-4fa0-4654-9fdc-50479bd52c50 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.314913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7175bc-117d-4a28-9962-7cb3d43161e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.315578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7175bc-117d-4a28-9962-7cb3d43161e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.315858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7175bc-117d-4a28-9962-7cb3d43161e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.316173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7175bc-117d-4a28-9962-7cb3d43161e3 admin admin] PATH_INFO: `/v3/projects/d3f9d50c0a794adf86cd27195b4eacef/users/63f40d05119c4951926b12fdf63404be/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.335794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7175bc-117d-4a28-9962-7cb3d43161e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3f9d50c0a794adf86cd27195b4eacef, user_id=63f40d05119c4951926b12fdf63404be, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.338648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7175bc-117d-4a28-9962-7cb3d43161e3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.346626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.348696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:21.354959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 899/1822] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:21 2022] PUT /identity/v3/projects/d3f9d50c0a794adf86cd27195b4eacef/users/63f40d05119c4951926b12fdf63404be/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:21.366305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f58d114-9b13-4c18-8226-17f81b97b025 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.366576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f58d114-9b13-4c18-8226-17f81b97b025 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.366847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f58d114-9b13-4c18-8226-17f81b97b025 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.368353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5520ec-13c3-4a39-8eab-80ccad52e372 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.401765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5f58d114-9b13-4c18-8226-17f81b97b025 None None] MFA Rules not processed for user `63f40d05119c4951926b12fdf63404be`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:21.402953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5f58d114-9b13-4c18-8226-17f81b97b025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:21.432319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.432935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.433236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.437501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.439808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:21.479012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 900/1823] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:21.483663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f58d114-9b13-4c18-8226-17f81b97b025 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.514305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.515762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.525695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 924/1824] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:21 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:21.566107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f58d114-9b13-4c18-8226-17f81b97b025 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.572451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.573332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.574038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.574317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.575173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.577534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.610866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 901/1825] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:21 2022] POST /identity/v3/projects => generated 365 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:21.615499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.626854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c605d70-b29e-4d74-bde7-3def7c1bdfb9 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.689556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.690268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.690548 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.690818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.691457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.693637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.722869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 925/1826] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:21 2022] POST /identity/v3/users => generated 289 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:21.727689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.740225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c705a361-903d-4e4e-b702-27f0e8f8e018 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.747194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.747950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.748097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.748329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.748986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.750597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.754743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:21.770034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 902/1827] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:21 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:21.817045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.817045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e882845-032d-40a3-87d6-010c20bfdece admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:21.817045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:21.817045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:21.833614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:21.833614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] PATH_INFO: `/v3/projects/4f85193c45f34b6cad6cedc67b102be6/users/5d5633ea0ebb476887e2f7e6c43f63c2/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:21.833614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f85193c45f34b6cad6cedc67b102be6, user_id=5d5633ea0ebb476887e2f7e6c43f63c2, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:21.833614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:21.918196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 926/1828] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:21 2022] PUT /identity/v3/projects/4f85193c45f34b6cad6cedc67b102be6/users/5d5633ea0ebb476887e2f7e6c43f63c2/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:21.920734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:21.932994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d86e5c1-833b-4c46-a50d-9be43325ac71 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:22.032257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80d8c09-78e2-4582-bb85-e96cc77c50e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:22.033203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d8c09-78e2-4582-bb85-e96cc77c50e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:22.033756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d8c09-78e2-4582-bb85-e96cc77c50e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:22.034006 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d8c09-78e2-4582-bb85-e96cc77c50e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:22.034830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80d8c09-78e2-4582-bb85-e96cc77c50e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:22.037279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80d8c09-78e2-4582-bb85-e96cc77c50e0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:22.048964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 903/1829] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:21 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:22.054679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:22.067303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:22.075156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34ffe93-20c9-4984-a73e-3cf4adbd658d admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:22.085745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:22.086852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:22.087234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:22.087684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] PATH_INFO: `/v3/projects/4f85193c45f34b6cad6cedc67b102be6/users/5d5633ea0ebb476887e2f7e6c43f63c2/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:22.091352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f85193c45f34b6cad6cedc67b102be6, user_id=5d5633ea0ebb476887e2f7e6c43f63c2, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:22.096615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:22.120062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 927/1830] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:22 2022] PUT /identity/v3/projects/4f85193c45f34b6cad6cedc67b102be6/users/5d5633ea0ebb476887e2f7e6c43f63c2/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:22.138909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5857f9-d73c-43f0-be32-46e82c6c8060 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:22.139193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5857f9-d73c-43f0-be32-46e82c6c8060 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:22.139465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5857f9-d73c-43f0-be32-46e82c6c8060 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:22.189849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8e5857f9-d73c-43f0-be32-46e82c6c8060 None None] MFA Rules not processed for user `5d5633ea0ebb476887e2f7e6c43f63c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:22.191088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8e5857f9-d73c-43f0-be32-46e82c6c8060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:22.289774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 904/1831] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:22 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:22.305608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:22.379774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6810aa-52ac-48ad-8c8b-e7d0ffeeacda admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:22.384995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:22.386033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:22.386508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:22.386948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:22.397811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:22.399232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:22.408592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 928/1832] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:22 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:22.476099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5857f9-d73c-43f0-be32-46e82c6c8060 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:22.487540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5857f9-d73c-43f0-be32-46e82c6c8060 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:22.491972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc3f464-b4ad-40fc-9f5d-b68e2fe6595e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:22.492583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc3f464-b4ad-40fc-9f5d-b68e2fe6595e service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:22.492857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc3f464-b4ad-40fc-9f5d-b68e2fe6595e service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:22.493184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc3f464-b4ad-40fc-9f5d-b68e2fe6595e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:22.563660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc3f464-b4ad-40fc-9f5d-b68e2fe6595e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:22.565048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc3f464-b4ad-40fc-9f5d-b68e2fe6595e service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:22.575743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 905/1833] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:22 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:26.361720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:26.449189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5542aa6-7834-4cad-b327-e9a223270326 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:26.455549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:26.456152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:26.456404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:26.456680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] PATH_INFO: `/v3/users/63f40d05119c4951926b12fdf63404be` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:26.458336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=63f40d05119c4951926b12fdf63404be)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:26.460950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:26.532406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63f40d05119c4951926b12fdf63404be'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:26.549037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63f40d05119c4951926b12fdf63404be'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:26.549896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63f40d05119c4951926b12fdf63404be'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:26.559283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63f40d05119c4951926b12fdf63404be'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:26.579701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63f40d05119c4951926b12fdf63404be'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:26.586890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 929/1834] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:26 2022] DELETE /identity/v3/users/63f40d05119c4951926b12fdf63404be => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:26.591981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:26.592526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:26.592902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:26.617986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:26.621041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:26.636014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:26.725214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 906/1835] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:26.736934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:26.738576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:26.760768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:26.820622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db8cafb-00cc-4e9c-9395-cfc4063335c0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:26.826072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:26.826749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:26.827088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:26.827243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:26.838821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:26.840258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:26.851621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 930/1836] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:27.232517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:27.235168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:27.271090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d9f65-f709-4ba0-9324-61bd76366d48 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:27.284959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:27.287942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:27.288232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:27.288499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] PATH_INFO: `/v3/projects/d3f9d50c0a794adf86cd27195b4eacef` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:27.291668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3f9d50c0a794adf86cd27195b4eacef)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:27.295756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:27.375310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3f9d50c0a794adf86cd27195b4eacef', 'request_id': 'req-77f76ed6-7619-4196-a2b3-6e4bb92838d0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.387455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:27.390446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:27.395979 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3f9d50c0a794adf86cd27195b4eacef', 'request_id': 'req-77f76ed6-7619-4196-a2b3-6e4bb92838d0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.405199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3f9d50c0a794adf86cd27195b4eacef', 'request_id': 'req-77f76ed6-7619-4196-a2b3-6e4bb92838d0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.414637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c97e74-1e4a-435f-8be7-e32688ddde46 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:27.421695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] The token cache is being invalidate because project d3f9d50c0a794adf86cd27195b4eacef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:27.422153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.423309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.425544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 907/1837] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:27 2022] DELETE /identity/v3/projects/d3f9d50c0a794adf86cd27195b4eacef => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:27.430667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:27.432825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:27.499416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:27.500001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:27.500338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:27.500804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] PATH_INFO: `/v3/users/9dbdca0071824603af2a1521eb2ea160` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:27.502251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9dbdca0071824603af2a1521eb2ea160)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:27.504311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:27.522625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f76ed6-7619-4196-a2b3-6e4bb92838d0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:27.527805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:27.528538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:27.528884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:27.529266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] PATH_INFO: `/v3/users/5d5633ea0ebb476887e2f7e6c43f63c2` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:27.530771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d5633ea0ebb476887e2f7e6c43f63c2)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:27.533246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:27.562657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9dbdca0071824603af2a1521eb2ea160'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.576813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9dbdca0071824603af2a1521eb2ea160'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.577649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9dbdca0071824603af2a1521eb2ea160'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.584653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:27.586472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9dbdca0071824603af2a1521eb2ea160'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.587478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d5633ea0ebb476887e2f7e6c43f63c2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.594619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a62ce359-1c4f-4cca-a0d7-708840f7c114 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9dbdca0071824603af2a1521eb2ea160'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.598686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d5633ea0ebb476887e2f7e6c43f63c2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.601676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 931/1838] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:27 2022] DELETE /identity/v3/users/9dbdca0071824603af2a1521eb2ea160 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:27.606342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:27.606621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:27.606894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:27.609129 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d5633ea0ebb476887e2f7e6c43f63c2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.609817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d5633ea0ebb476887e2f7e6c43f63c2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.610666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d5633ea0ebb476887e2f7e6c43f63c2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:27.627378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:27.628611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:27.633091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 908/1839] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:27 2022] DELETE /identity/v3/users/5d5633ea0ebb476887e2f7e6c43f63c2 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:27.666924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:27.704055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 932/1840] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:27.719741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:27.723207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:27.825169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1fa3b9-fdcb-48f5-acab-9c4807d3b549 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:27.829657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:27.830505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:27.830782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:27.831052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:27.926327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:27.929990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:27.943668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 909/1841] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:28.100709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:28.102229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.126024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f824d211-802c-4fbc-812b-cddd86f8196d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:28.137796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:28.139177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.141022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.141625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] PATH_INFO: `/v3/projects/4f85193c45f34b6cad6cedc67b102be6` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.150482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f85193c45f34b6cad6cedc67b102be6)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:28.157165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:28.196361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:28.200418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.217274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c476e1-8d2a-4a0d-862e-9b3c7a254eca service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:28.225842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:28.227021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.227344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.227617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] PATH_INFO: `/v3/users/fe6fd08203e2402585695d3c51e92b7a` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.229700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe6fd08203e2402585695d3c51e92b7a)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:28.232598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:28.251594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f85193c45f34b6cad6cedc67b102be6', 'request_id': 'req-9044c821-ba46-4fc0-9b69-6a210a36a557'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.262120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f85193c45f34b6cad6cedc67b102be6', 'request_id': 'req-9044c821-ba46-4fc0-9b69-6a210a36a557'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.271880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f85193c45f34b6cad6cedc67b102be6', 'request_id': 'req-9044c821-ba46-4fc0-9b69-6a210a36a557'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.290599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] The token cache is being invalidate because project 4f85193c45f34b6cad6cedc67b102be6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:28.290982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.291696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9044c821-ba46-4fc0-9b69-6a210a36a557 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.299459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 933/1842] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:28 2022] DELETE /identity/v3/projects/4f85193c45f34b6cad6cedc67b102be6 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:28.302192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe6fd08203e2402585695d3c51e92b7a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.304566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046af202-1ea1-40c3-a759-d7e8fbd79bdb None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.304854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046af202-1ea1-40c3-a759-d7e8fbd79bdb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.305189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046af202-1ea1-40c3-a759-d7e8fbd79bdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.312747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe6fd08203e2402585695d3c51e92b7a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.324568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe6fd08203e2402585695d3c51e92b7a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.328238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe6fd08203e2402585695d3c51e92b7a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.329149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe6fd08203e2402585695d3c51e92b7a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.334557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:28.367152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-046af202-1ea1-40c3-a759-d7e8fbd79bdb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:28.368390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-046af202-1ea1-40c3-a759-d7e8fbd79bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.369944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 910/1843] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:28 2022] DELETE /identity/v3/users/fe6fd08203e2402585695d3c51e92b7a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:28.375179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:28.377186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.439458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 934/1844] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:28.441442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84794426-7d31-48c8-883b-a9009a59c3c8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.441890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84794426-7d31-48c8-883b-a9009a59c3c8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.442395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84794426-7d31-48c8-883b-a9009a59c3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.467967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-84794426-7d31-48c8-883b-a9009a59c3c8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:28.470102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-84794426-7d31-48c8-883b-a9009a59c3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.482642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 935/1845] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:28.484417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.484929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.485364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.486287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f1f804-4f17-4321-86ef-49450a45fa80 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:28.494842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:28.495660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.496048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.496415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] PATH_INFO: `/v3/projects/2405358a6fa84b4e99afe6864eed5eaf` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.498706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2405358a6fa84b4e99afe6864eed5eaf)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:28.501601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:28.538093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:28.542952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.560689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 936/1846] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:28.562194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:28.565301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.582650 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2405358a6fa84b4e99afe6864eed5eaf', 'request_id': 'req-946c30d2-c370-4c24-aecf-715d796b2d8b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.599775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2405358a6fa84b4e99afe6864eed5eaf', 'request_id': 'req-946c30d2-c370-4c24-aecf-715d796b2d8b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.607263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2405358a6fa84b4e99afe6864eed5eaf', 'request_id': 'req-946c30d2-c370-4c24-aecf-715d796b2d8b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.628535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] The token cache is being invalidate because project 2405358a6fa84b4e99afe6864eed5eaf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:28.629054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.630248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.632581 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 911/1847] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:28 2022] DELETE /identity/v3/projects/2405358a6fa84b4e99afe6864eed5eaf => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:28.633807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:28.636114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.646978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:28.670347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1534f9f-a1e0-4d45-8cc9-968a4dccda9e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:28.731339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c30d2-c370-4c24-aecf-715d796b2d8b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:28.747507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:28.748576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.748576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.748867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.749423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:28.749730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:28.751354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:28.753975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c6cd09-8b7f-4208-bbbe-6bbc53bdf22c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:28.754593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c6cd09-8b7f-4208-bbbe-6bbc53bdf22c service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.756600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c6cd09-8b7f-4208-bbbe-6bbc53bdf22c service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.756879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c6cd09-8b7f-4208-bbbe-6bbc53bdf22c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.760390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c3c6cd09-8b7f-4208-bbbe-6bbc53bdf22c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.771713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 912/1848] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:28.773351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:28.777703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:28.780218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c6cd09-8b7f-4208-bbbe-6bbc53bdf22c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:28.782305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c6cd09-8b7f-4208-bbbe-6bbc53bdf22c service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:28.804682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 937/1849] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:28.806422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692662e4-5bfc-47e4-8dc6-9d73fc5774cf None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.806755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692662e4-5bfc-47e4-8dc6-9d73fc5774cf None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.807365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692662e4-5bfc-47e4-8dc6-9d73fc5774cf None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.808526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 938/1850] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:23:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:23:28.809409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c13ea0-e29c-496f-9551-beebdc60a81d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:28.815415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692662e4-5bfc-47e4-8dc6-9d73fc5774cf None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:28.816248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:28.816477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.816757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.817239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] PATH_INFO: `/v3/users/ab9be73292e34daa829b42d51d7d3f89` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.820057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab9be73292e34daa829b42d51d7d3f89)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:28.821933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:28.824004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:28.846080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692662e4-5bfc-47e4-8dc6-9d73fc5774cf None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:28.853387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb32951e-e215-4f26-b070-6f533050eac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:28.855101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32951e-e215-4f26-b070-6f533050eac4 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.855592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32951e-e215-4f26-b070-6f533050eac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.856063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32951e-e215-4f26-b070-6f533050eac4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.857263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb32951e-e215-4f26-b070-6f533050eac4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:28.859409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb32951e-e215-4f26-b070-6f533050eac4 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:28.921211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 939/1851] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:28 2022] POST /identity/v3/projects => generated 389 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:28.921211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:28.921211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:28.921596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:28.925172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab9be73292e34daa829b42d51d7d3f89'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.935163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab9be73292e34daa829b42d51d7d3f89'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.952627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab9be73292e34daa829b42d51d7d3f89'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.953612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab9be73292e34daa829b42d51d7d3f89'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.954535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab9be73292e34daa829b42d51d7d3f89'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:28.956195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:28.959397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.015426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 913/1852] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:28 2022] DELETE /identity/v3/users/ab9be73292e34daa829b42d51d7d3f89 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:29.016294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.037757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.153721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 940/1853] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:23:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:29.155383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.158065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.234224 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f99bb7-ff74-4d65-91b8-78eacc2c3612 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.244916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.245479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.245938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.246391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.248464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.252614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.306173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 914/1854] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:29 2022] POST /identity/v3/users => generated 295 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:29.307692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.310391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.337088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82ef568-2d16-42ba-bd8b-e73d92bffa8a None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.348666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.351064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.351378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.352006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] PATH_INFO: `/v3/users/780f06ac3efb41b188805de61d79b372` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.356825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] RBAC: Authorizing `identity:delete_user(user_id=780f06ac3efb41b188805de61d79b372)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.360609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.408068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:29.488380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908659b6-337b-4c0c-81a9-116da84085fd admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.505867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.506850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.507363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.507949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.519414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:29.527751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '780f06ac3efb41b188805de61d79b372'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.529285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.533358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.564412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 915/1855] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:29.565447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.577175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '780f06ac3efb41b188805de61d79b372'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.579584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '780f06ac3efb41b188805de61d79b372'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.582711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543052f5-386c-4ff7-8f78-91e907eae576 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.592042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '780f06ac3efb41b188805de61d79b372'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.596896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.597515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.597791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.598284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.599027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.603342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.609961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '780f06ac3efb41b188805de61d79b372'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.615797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 916/1856] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:29 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:29.618657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.621373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 941/1857] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:29 2022] DELETE /identity/v3/users/780f06ac3efb41b188805de61d79b372 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:29.621844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.622744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.624223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.641766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3769ed39-17c1-4e8c-9e65-721929018d41 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.707157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.708499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.708632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.708805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887aa7eb-9f0a-4618-92f9-62b8b348d255 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.709058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] PATH_INFO: `/v3/projects/d2911f74871042e5ac518303c21bde26` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.710791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d2911f74871042e5ac518303c21bde26)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.712758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.721257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.722877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.722877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.722877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] PATH_INFO: `/v3/projects/698cec9c23d94e5389cb740a61096b4b` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.723927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=698cec9c23d94e5389cb740a61096b4b)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.725645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.774583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd2911f74871042e5ac518303c21bde26', 'request_id': 'req-178f34eb-cbe7-4524-959a-ceb0dd09f705'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.778860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '698cec9c23d94e5389cb740a61096b4b', 'request_id': 'req-823596cf-0f61-4510-a9d7-4a49c1230c79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.785991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '698cec9c23d94e5389cb740a61096b4b', 'request_id': 'req-823596cf-0f61-4510-a9d7-4a49c1230c79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.788025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd2911f74871042e5ac518303c21bde26', 'request_id': 'req-178f34eb-cbe7-4524-959a-ceb0dd09f705'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.792921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '698cec9c23d94e5389cb740a61096b4b', 'request_id': 'req-823596cf-0f61-4510-a9d7-4a49c1230c79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.795058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd2911f74871042e5ac518303c21bde26', 'request_id': 'req-178f34eb-cbe7-4524-959a-ceb0dd09f705'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.805955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] The token cache is being invalidate because project 698cec9c23d94e5389cb740a61096b4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:29.806361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.807018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.808408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] The token cache is being invalidate because project d2911f74871042e5ac518303c21bde26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:29.808547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 942/1858] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:29 2022] DELETE /identity/v3/projects/698cec9c23d94e5389cb740a61096b4b => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:29.809232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.809514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.809950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-178f34eb-cbe7-4524-959a-ceb0dd09f705 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:29.810866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.811952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 917/1859] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:29 2022] DELETE /identity/v3/projects/d2911f74871042e5ac518303c21bde26 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:29.814395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd654d6c-1d36-4551-afab-4215793a91ac None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.814945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd654d6c-1d36-4551-afab-4215793a91ac None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.815225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd654d6c-1d36-4551-afab-4215793a91ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.835303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-bd654d6c-1d36-4551-afab-4215793a91ac None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:29.836532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bd654d6c-1d36-4551-afab-4215793a91ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.840676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:29.880797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823596cf-0f61-4510-a9d7-4a49c1230c79 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.886060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.886674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.886958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.887229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.888111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.889648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.903773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 943/1860] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:29 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:29.907877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 918/1861] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:29.909322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.909610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.909940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.914550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.918595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.939681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af8e44-88d6-49eb-a37e-e090f97f09e8 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.941329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:29.943539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.946013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.946980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.947362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.947709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] PATH_INFO: `/v3/users/eb5dae33b5fc401caed59579c721736e` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.949289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb5dae33b5fc401caed59579c721736e)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:29.951952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:29.959439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 919/1862] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:29.961150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:29.963802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:29.982661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21021bd-7fe0-4734-9a62-f92b69c9a835 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:29.990924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:29.991767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:29.992252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:29.992735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] PATH_INFO: `/v3/projects/478b5c9f564f423b9670ec81740a5855/users/b7e3da5d9f074ecfb2aa7fd57e56190f/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:29.996178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:30.011973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb5dae33b5fc401caed59579c721736e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:30.022112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=478b5c9f564f423b9670ec81740a5855, user_id=b7e3da5d9f074ecfb2aa7fd57e56190f, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.028480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.036461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb5dae33b5fc401caed59579c721736e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:30.038680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb5dae33b5fc401caed59579c721736e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:30.049671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb5dae33b5fc401caed59579c721736e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:30.052960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 920/1863] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:29 2022] PUT /identity/v3/projects/478b5c9f564f423b9670ec81740a5855/users/b7e3da5d9f074ecfb2aa7fd57e56190f/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:30.053688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.055358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:30.074165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fd583d4-a905-4cfe-b239-336bf8b5c297 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb5dae33b5fc401caed59579c721736e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:30.086173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 944/1864] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:29 2022] DELETE /identity/v3/users/eb5dae33b5fc401caed59579c721736e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:30.086173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246937fe-f0ab-4607-b71e-61387b367196 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.086173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246937fe-f0ab-4607-b71e-61387b367196 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.086605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246937fe-f0ab-4607-b71e-61387b367196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.122406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-246937fe-f0ab-4607-b71e-61387b367196 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:30.127391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-246937fe-f0ab-4607-b71e-61387b367196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:30.158490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614c1e2d-bf74-4c04-8e19-67c8de60fcd1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.217299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 945/1865] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:30.218469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.218758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.219023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.254224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] MFA Rules not processed for user `b7e3da5d9f074ecfb2aa7fd57e56190f`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:30.255582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:30.257070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.257896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.258179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.258514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.260759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:30.276308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.277795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.281915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:30.291660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 921/1866] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:30.293192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.312667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499bc05d-a5c9-40ef-a862-8e532ebf8b59 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.320885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.321835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.322346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.322817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.324330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:30.325269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.326769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.350498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 922/1867] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:30.356188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 946/1868] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:30 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:30.356936 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.361473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.362783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:30.380723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9782a59-af2f-4fad-bd4e-1cb8911f6fe8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.383798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2a5c4-291c-4a65-af81-bb171a1db688 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.390119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.390923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.391090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.391701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.392647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.392763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.393383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.393526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.394199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.395576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.396478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.399513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.439505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 947/1869] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:30 2022] POST /identity/v3/projects => generated 387 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:30.444210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.455906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 923/1870] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:30 2022] POST /identity/v3/projects => generated 369 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:30.462326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.565185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70973743-36da-4d1c-b636-0fcc7d6417aa admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.603963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d185cfb6-2397-4979-8d73-1894619ad903 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.610997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.612145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.612631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.613097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.614316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.616642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.636174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:30.665876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 924/1871] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:30 2022] POST /identity/v3/users => generated 285 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:30.665876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.674188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.675163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.676217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.677247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.679530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.683313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.683826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a50e2d-3b61-4c21-bf6f-dfcba0f1e887 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.696688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.697349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.697627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.697942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.698876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.704000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.733797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 925/1872] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:30 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:30.734239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.740348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 948/1873] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:30 2022] POST /identity/v3/users => generated 300 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:30.745928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.754550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14793aea-6d35-4988-a0b1-da08220038ed admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.769654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.770772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.771117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.771712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.772327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c68785-6666-4afe-9ea7-c853f8aed550 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.772648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.778901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.782891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.784317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.785212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.786027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.787955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.792394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.815087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 926/1874] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:30 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:30.816430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.830151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 949/1875] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:30 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:30.834503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.843592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff9fa58-1d8f-4a67-9337-cb95c4b97e6a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.860919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.862148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.867276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.868260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] PATH_INFO: `/v3/projects/616d8c6feff34df68117b541eb6b6ecd/users/1d542c14c4b149be8ee26c83697d22f9/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.869649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfc7862-3207-472a-a75e-89fcc5a5b6ba admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:30.891354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3641feca-1e74-4059-b617-b79d2e9b5bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:30.894377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641feca-1e74-4059-b617-b79d2e9b5bf2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:30.905537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641feca-1e74-4059-b617-b79d2e9b5bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:30.905537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641feca-1e74-4059-b617-b79d2e9b5bf2 admin admin] PATH_INFO: `/v3/projects/8e970e8e99d041e6bc97a47d4e43fc50/users/44bf0938e63940029dbc092c15734caa/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:30.922471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=616d8c6feff34df68117b541eb6b6ecd, user_id=1d542c14c4b149be8ee26c83697d22f9, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.945789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.972416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3641feca-1e74-4059-b617-b79d2e9b5bf2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e970e8e99d041e6bc97a47d4e43fc50, user_id=44bf0938e63940029dbc092c15734caa, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:30.975669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3641feca-1e74-4059-b617-b79d2e9b5bf2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:30.986434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 927/1876] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:30 2022] PUT /identity/v3/projects/616d8c6feff34df68117b541eb6b6ecd/users/1d542c14c4b149be8ee26c83697d22f9/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:30.987928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:30.994821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:31.014696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 950/1877] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:30 2022] PUT /identity/v3/projects/8e970e8e99d041e6bc97a47d4e43fc50/users/44bf0938e63940029dbc092c15734caa/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:31.019919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d881aa-0ecb-43c7-97a6-f21dd698b7ac None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:31.020437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d881aa-0ecb-43c7-97a6-f21dd698b7ac None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:31.025189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d881aa-0ecb-43c7-97a6-f21dd698b7ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:31.037336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:31.072716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-55d881aa-0ecb-43c7-97a6-f21dd698b7ac None None] MFA Rules not processed for user `1d542c14c4b149be8ee26c83697d22f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:31.075393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-55d881aa-0ecb-43c7-97a6-f21dd698b7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:31.134858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3381a9b3-7290-4016-bac6-63a92bca65aa admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:31.179558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 951/1878] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:31 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:31.181082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d881aa-0ecb-43c7-97a6-f21dd698b7ac None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:31.244388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:31.246250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:31.247426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:31.248039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] PATH_INFO: `/v3/projects/331d4970745b449bad65671bec6351c8` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:31.252889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=331d4970745b449bad65671bec6351c8)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:31.260074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:31.290191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d881aa-0ecb-43c7-97a6-f21dd698b7ac None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:31.304084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:31.304809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:31.305177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:31.305519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:31.307623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:31.309374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:31.336597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 952/1879] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:31 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:31.337624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:31.354276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f12320-b632-45ea-84e6-1a0cac011d35 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:31.389427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '331d4970745b449bad65671bec6351c8', 'request_id': 'req-628d9590-8a6e-4234-a566-5fc65acacb3f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:31.402747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '331d4970745b449bad65671bec6351c8', 'request_id': 'req-628d9590-8a6e-4234-a566-5fc65acacb3f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:31.406094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:31.419248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '331d4970745b449bad65671bec6351c8', 'request_id': 'req-628d9590-8a6e-4234-a566-5fc65acacb3f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:31.443440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] The token cache is being invalidate because project 331d4970745b449bad65671bec6351c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:31.443713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:31.445238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:31.450389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 928/1880] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:30 2022] DELETE /identity/v3/projects/331d4970745b449bad65671bec6351c8 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:31.456044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:31.459971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:31.492042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:31.492666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:31.492943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:31.493305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:31.494021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:31.496105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:31.533451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 953/1881] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:31 2022] POST /identity/v3/projects => generated 369 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:31.534396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:31.536616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:31.554427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:31.622535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06bf80-a200-4694-a3dd-4a92e77c80a6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:31.630539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:31.631491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:31.631994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:31.632377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] PATH_INFO: `/v3/projects/8e970e8e99d041e6bc97a47d4e43fc50/users/44bf0938e63940029dbc092c15734caa/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:31.634652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e970e8e99d041e6bc97a47d4e43fc50, user_id=44bf0938e63940029dbc092c15734caa, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:31.638784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:31.652241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 954/1882] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:31 2022] PUT /identity/v3/projects/8e970e8e99d041e6bc97a47d4e43fc50/users/44bf0938e63940029dbc092c15734caa/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:31.653470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:31.666485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:31.696131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628d9590-8a6e-4234-a566-5fc65acacb3f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:31.806243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f1159-7216-4846-af84-eb1b0b524284 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:31.821999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:31.822702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:31.825006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:31.825676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] PATH_INFO: `/v3/users/8562555379ec44aa8ebfd83ec0a1eb78` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:31.826825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8562555379ec44aa8ebfd83ec0a1eb78)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:31.834878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:31.936705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:31.937845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:31.938488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:31.938858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] PATH_INFO: `/v3/projects/6904bf36e551422b862effe3e98b85f6` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:31.940742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6904bf36e551422b862effe3e98b85f6)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:31.944509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:31.955702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8562555379ec44aa8ebfd83ec0a1eb78'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:31.994872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8562555379ec44aa8ebfd83ec0a1eb78'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:31.995646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8562555379ec44aa8ebfd83ec0a1eb78'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.023623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8562555379ec44aa8ebfd83ec0a1eb78'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.035468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8562555379ec44aa8ebfd83ec0a1eb78'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.053553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 955/1883] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:31 2022] DELETE /identity/v3/users/8562555379ec44aa8ebfd83ec0a1eb78 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:32.054767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:32.056485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:32.079671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6904bf36e551422b862effe3e98b85f6', 'request_id': 'req-de8fca90-a266-4c39-b7db-aa95d3f5c255'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.105937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6904bf36e551422b862effe3e98b85f6', 'request_id': 'req-de8fca90-a266-4c39-b7db-aa95d3f5c255'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.116467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6904bf36e551422b862effe3e98b85f6', 'request_id': 'req-de8fca90-a266-4c39-b7db-aa95d3f5c255'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.134556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] The token cache is being invalidate because project 6904bf36e551422b862effe3e98b85f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:32.135068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.136557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-de8fca90-a266-4c39-b7db-aa95d3f5c255 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.139938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 929/1884] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:31 2022] DELETE /identity/v3/projects/6904bf36e551422b862effe3e98b85f6 => generated 0 bytes in 689 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:32.143112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:32.143670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:32.144004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:32.152632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:32.206893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] MFA Rules not processed for user `44bf0938e63940029dbc092c15734caa`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:32.214741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:32.229285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5e20c-1462-4b09-b366-606e9ddc58e5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:32.243637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:32.244724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:32.245528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:32.246067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:32.246953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:32.249228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:32.261798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:32.289656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 956/1885] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:32 2022] POST /identity/v3/users => generated 291 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:32.290602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:32.301007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:32.337538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324a7e76-318c-4387-8171-ded841aea38e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:32.356294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:32.358563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:32.359563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:32.363432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] PATH_INFO: `/v3/users/d38ffa2630af41ff848e98418e551057` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:32.365766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d38ffa2630af41ff848e98418e551057)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:32.367724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:32.404096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 930/1886] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:32 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:32.405475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:32.409353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:32.440716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14c068b-3315-4735-969d-63a1c0138db6 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:32.456578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:32.459392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:32.459719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:32.461206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:32.462111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:32.463964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:32.476588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd38ffa2630af41ff848e98418e551057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.494468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 931/1887] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:32 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:32.498698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd38ffa2630af41ff848e98418e551057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.499614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:32.500269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd38ffa2630af41ff848e98418e551057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.501686 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:32.512678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd38ffa2630af41ff848e98418e551057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.527135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd38ffa2630af41ff848e98418e551057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:32.536442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 957/1888] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:32 2022] DELETE /identity/v3/users/d38ffa2630af41ff848e98418e551057 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:32.537486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:32.538923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:32.591759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4cc076-82f8-407d-ab51-9c03be8199c9 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:32.753723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afb30f5-dfcc-49e1-9134-badc3b6b6f1a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:32.759667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:32.760920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:32.760920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:32.761087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:32.763453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:32.771812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:32.772596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:32.772879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:32.773174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] PATH_INFO: `/v3/projects/6251924d8732447fb5e5f021836e1ffd/users/c51424434ab84373b959a1b14cf46091/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:32.836063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6251924d8732447fb5e5f021836e1ffd, user_id=c51424434ab84373b959a1b14cf46091, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:32.848741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:32.879191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 958/1889] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:32 2022] PUT /identity/v3/projects/6251924d8732447fb5e5f021836e1ffd/users/c51424434ab84373b959a1b14cf46091/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:32.883462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:32.916262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:32.961909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:32.964655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:32.980376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 932/1890] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:32 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 485 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:32.992862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:32.998707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:33.018383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:33.043740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7683e6ec-9f73-47ed-8de5-9922cc40ee62 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:33.053214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:33.054689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.058527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78152b06-d42d-43c1-84f6-f69280942064 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:33.063725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:33.064341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.064820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.065025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.065664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:33.067689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:33.068593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.071661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] PATH_INFO: `/v3/projects/73184ddaf13f4353b1de04407b614d99` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.075601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=73184ddaf13f4353b1de04407b614d99)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:33.077943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:33.092480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 959/1891] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:32 2022] GET /identity/v3/roles => generated 1420 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:33.093452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:33.107698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30a4e87-f346-4ae1-b816-4f93a4000122 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:33.123151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c147a498-33d9-4907-9a6a-3e8f3133c698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:33.123849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c147a498-33d9-4907-9a6a-3e8f3133c698 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.124121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c147a498-33d9-4907-9a6a-3e8f3133c698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.124392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c147a498-33d9-4907-9a6a-3e8f3133c698 admin admin] PATH_INFO: `/v3/projects/6251924d8732447fb5e5f021836e1ffd/users/c51424434ab84373b959a1b14cf46091/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.130395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c147a498-33d9-4907-9a6a-3e8f3133c698 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6251924d8732447fb5e5f021836e1ffd, user_id=c51424434ab84373b959a1b14cf46091, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:33.134229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c147a498-33d9-4907-9a6a-3e8f3133c698 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:33.174333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 960/1892] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:33 2022] PUT /identity/v3/projects/6251924d8732447fb5e5f021836e1ffd/users/c51424434ab84373b959a1b14cf46091/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:33.194830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.196856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.196856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.230748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73184ddaf13f4353b1de04407b614d99', 'request_id': 'req-06d23808-266e-4fc5-ba44-61fe76c4c7fc'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.249968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73184ddaf13f4353b1de04407b614d99', 'request_id': 'req-06d23808-266e-4fc5-ba44-61fe76c4c7fc'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.291681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73184ddaf13f4353b1de04407b614d99', 'request_id': 'req-06d23808-266e-4fc5-ba44-61fe76c4c7fc'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.303812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] MFA Rules not processed for user `c51424434ab84373b959a1b14cf46091`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:33.308674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:33.331772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] The token cache is being invalidate because project 73184ddaf13f4353b1de04407b614d99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:33.332161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.333012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.346115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 933/1893] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:32 2022] DELETE /identity/v3/projects/73184ddaf13f4353b1de04407b614d99 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:33.346115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:33.349080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:33.495429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 961/1894] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:33 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:33.497033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:33.498188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:33.511056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d23808-266e-4fc5-ba44-61fe76c4c7fc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:33.616983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:33.618045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.618500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.619183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] PATH_INFO: `/v3/projects/37f8fe6f87be453baaaca4c47bc6a0f7` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.623140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] RBAC: Authorizing `identity:delete_project(project_id=37f8fe6f87be453baaaca4c47bc6a0f7)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:33.629158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39c3b63-538b-4e13-9a7c-652dd4ccdd10 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:33.630897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:33.640238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:33.641204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.641654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.642254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] PATH_INFO: `/v3/users/b7e3da5d9f074ecfb2aa7fd57e56190f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.644478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7e3da5d9f074ecfb2aa7fd57e56190f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:33.648298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:33.702580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:33.723893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:33.743445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37f8fe6f87be453baaaca4c47bc6a0f7', 'request_id': 'req-405f6c42-528c-4f51-a63a-502cf3a0389b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.758505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37f8fe6f87be453baaaca4c47bc6a0f7', 'request_id': 'req-405f6c42-528c-4f51-a63a-502cf3a0389b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.768109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37f8fe6f87be453baaaca4c47bc6a0f7', 'request_id': 'req-405f6c42-528c-4f51-a63a-502cf3a0389b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.769171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7e3da5d9f074ecfb2aa7fd57e56190f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.786191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7e3da5d9f074ecfb2aa7fd57e56190f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.787429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7e3da5d9f074ecfb2aa7fd57e56190f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.792905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] The token cache is being invalidate because project 37f8fe6f87be453baaaca4c47bc6a0f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:33.794628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.796435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-405f6c42-528c-4f51-a63a-502cf3a0389b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.799818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 934/1895] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:33 2022] DELETE /identity/v3/projects/37f8fe6f87be453baaaca4c47bc6a0f7 => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:33.800845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7e3da5d9f074ecfb2aa7fd57e56190f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.801083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c063ad5-2e61-41cb-884f-6ed394f44d26 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.801388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c063ad5-2e61-41cb-884f-6ed394f44d26 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.802049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c063ad5-2e61-41cb-884f-6ed394f44d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.822628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7e3da5d9f074ecfb2aa7fd57e56190f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:33.828366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1c063ad5-2e61-41cb-884f-6ed394f44d26 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:33.830028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1c063ad5-2e61-41cb-884f-6ed394f44d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:33.842303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 962/1896] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:33 2022] DELETE /identity/v3/users/b7e3da5d9f074ecfb2aa7fd57e56190f => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:33.844485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:33.851008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:33.930416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 935/1897] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:33.930416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85ef33-3bde-4d53-a6a5-3d93af210c59 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.930416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85ef33-3bde-4d53-a6a5-3d93af210c59 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.930416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f85ef33-3bde-4d53-a6a5-3d93af210c59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.937341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a1f1cd-d61c-4d1a-8325-d3658e32a6f3 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:33.939814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca341d2-4e37-4dc2-9504-1cc794a4b25b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:33.940563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca341d2-4e37-4dc2-9504-1cc794a4b25b service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:33.940905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca341d2-4e37-4dc2-9504-1cc794a4b25b service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:33.941274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca341d2-4e37-4dc2-9504-1cc794a4b25b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:33.943416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-eca341d2-4e37-4dc2-9504-1cc794a4b25b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:33.948289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1f85ef33-3bde-4d53-a6a5-3d93af210c59 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:33.950008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1f85ef33-3bde-4d53-a6a5-3d93af210c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.014703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca341d2-4e37-4dc2-9504-1cc794a4b25b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.016144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca341d2-4e37-4dc2-9504-1cc794a4b25b service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.016409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 936/1898] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:34.017815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdedc32e-cda7-46ec-9b81-c99d0a6a58cb None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.018208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdedc32e-cda7-46ec-9b81-c99d0a6a58cb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.018474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdedc32e-cda7-46ec-9b81-c99d0a6a58cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.030040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 963/1899] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:33 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:34.031318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.031318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.031465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.048045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-fdedc32e-cda7-46ec-9b81-c99d0a6a58cb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:34.049548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fdedc32e-cda7-46ec-9b81-c99d0a6a58cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.060008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 937/1900] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:34.061502 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a73b7-a907-4257-a7a9-2cb2c07e7728 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.061801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a73b7-a907-4257-a7a9-2cb2c07e7728 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.062101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a73b7-a907-4257-a7a9-2cb2c07e7728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.064819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:34.066292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.077038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 964/1901] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:34.077987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.079394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.085772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-244a73b7-a907-4257-a7a9-2cb2c07e7728 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:34.087425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-244a73b7-a907-4257-a7a9-2cb2c07e7728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.101303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 938/1902] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:34.102641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a73b7-a907-4257-a7a9-2cb2c07e7728 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.117080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a73b7-a907-4257-a7a9-2cb2c07e7728 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.125063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.125812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.126087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.126353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.127041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:34.127341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.129178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.143118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 939/1903] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:34.148237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.149343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:34.158859 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d0afea-4dc9-42f3-8480-9e011a5cd283 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.164019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.165015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.165015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.165364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.165949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:34.166251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.167902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.180869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32781efb-7ff7-4cd7-8112-d4fa9da9b332 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.184323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.185071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.185521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.185892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.186964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 940/1904] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:34.191306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.196976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.199475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.203186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ff2b5-fb2e-4f2a-a3bb-2bafb7b385e7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.209315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.210156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.210354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.216341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.216341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.216341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.217309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 965/1905] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:34.218725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.249034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ae9e3-15dd-44a4-9558-1e7465148be3 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.263808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.265770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 941/1906] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/projects => generated 371 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:34.268839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.269140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.269430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.270078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.272089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.272223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.282170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:34.332263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 966/1907] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/projects => generated 383 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:34.339663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.380263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d44633-2ed5-4ff0-9d6f-ee01567c7c24 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.488665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e34ad3e-0257-4d76-9561-9baaac0d119f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.497218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.497908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.498257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.498539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.499869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.502595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.537061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.537857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.538138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.538607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.539265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.541269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.555471 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 967/1908] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/users => generated 292 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:34.563190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.576557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04208076-7114-4aea-b220-1c2945cd4164 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.582119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 942/1909] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/users => generated 286 bytes in 307 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:34.583694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.584342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.584596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.584937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.585625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.587387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.587508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.601157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 968/1910] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:34 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:34.605242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.606411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108892fc-de15-4eb3-8585-c4ada2ce779a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.615440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.616652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.617013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.617347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.618299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.619369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73180c-875f-4107-9ab9-a9b1a909ffd6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.620437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.626599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.627826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.628150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.628489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.629770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.631855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.633616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 943/1911] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:34 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:34.637783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.656242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44495a7f-1793-4f6d-950f-ffef74be83e9 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.657671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 969/1912] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:34 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:34.657671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.663317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.675112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.676027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.676187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.676546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.677297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.679135 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.680604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd40d952-a2e3-4c3c-984e-b9d47feac172 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.687649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.688578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.688857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.689149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] PATH_INFO: `/v3/projects/478b5c9f564f423b9670ec81740a5855` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.691425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=478b5c9f564f423b9670ec81740a5855)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.694120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.700024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 944/1913] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:34 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:34.701029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.746432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e5098f-d133-40ca-b208-a5abfdc9cbda admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.758664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.759433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.760349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.760608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] PATH_INFO: `/v3/projects/f1683bdac0ee4862861118a5a4bd9996/users/873ecd30f8984b8ca7a6f58c829b0f4f/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.774021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '478b5c9f564f423b9670ec81740a5855', 'request_id': 'req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:34.781729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1683bdac0ee4862861118a5a4bd9996, user_id=873ecd30f8984b8ca7a6f58c829b0f4f, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.782315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '478b5c9f564f423b9670ec81740a5855', 'request_id': 'req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:34.795682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '478b5c9f564f423b9670ec81740a5855', 'request_id': 'req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:34.795872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.811748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] The token cache is being invalidate because project 478b5c9f564f423b9670ec81740a5855 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:34.812036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:34.812884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:34.814653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 970/1914] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:34 2022] DELETE /identity/v3/projects/478b5c9f564f423b9670ec81740a5855 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:34.815739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.817770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.831216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 945/1915] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:34 2022] PUT /identity/v3/projects/f1683bdac0ee4862861118a5a4bd9996/users/873ecd30f8984b8ca7a6f58c829b0f4f/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:34.833441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:34.838114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:34.844996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:34.935362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932765ec-8e3c-4c58-ab23-a67e54e2d8e7 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.938058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:34.941806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbc12d9-9bb1-49e9-8b60-517d2fca9741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.942625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc12d9-9bb1-49e9-8b60-517d2fca9741 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.942956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc12d9-9bb1-49e9-8b60-517d2fca9741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.943187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc12d9-9bb1-49e9-8b60-517d2fca9741 admin admin] PATH_INFO: `/v3/projects/46f05e0ab37c4e52afb82496e4554101/users/0d0911420819469e83dc393a41f1a66f/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.946486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6fb56d-90f2-4045-9d66-18034c44a62e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:34.969046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbc12d9-9bb1-49e9-8b60-517d2fca9741 admin admin] RBAC: Authorizing `identity:create_grant(project_id=46f05e0ab37c4e52afb82496e4554101, user_id=0d0911420819469e83dc393a41f1a66f, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.971793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:34.972458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.972741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.972902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbc12d9-9bb1-49e9-8b60-517d2fca9741 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.973035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] PATH_INFO: `/v3/users/44bf0938e63940029dbc092c15734caa` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:34.974856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44bf0938e63940029dbc092c15734caa)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:34.977454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:34.997029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 971/1916] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:34 2022] PUT /identity/v3/projects/46f05e0ab37c4e52afb82496e4554101/users/0d0911420819469e83dc393a41f1a66f/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:34.998771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0f186-fa94-42cf-9ca0-00bfa48e7086 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:34.999513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0f186-fa94-42cf-9ca0-00bfa48e7086 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:34.999897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0f186-fa94-42cf-9ca0-00bfa48e7086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.043347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-eea0f186-fa94-42cf-9ca0-00bfa48e7086 None None] MFA Rules not processed for user `873ecd30f8984b8ca7a6f58c829b0f4f`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:35.044843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-eea0f186-fa94-42cf-9ca0-00bfa48e7086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:35.079090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44bf0938e63940029dbc092c15734caa'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.093713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44bf0938e63940029dbc092c15734caa'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.111573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44bf0938e63940029dbc092c15734caa'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.112463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44bf0938e63940029dbc092c15734caa'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.113363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44bf0938e63940029dbc092c15734caa'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.153903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 972/1917] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:35.157382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.157382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.157382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.157756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 946/1918] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:34 2022] DELETE /identity/v3/users/44bf0938e63940029dbc092c15734caa => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:35.160048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:35.163548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:35.194058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:35.195015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:35.277742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f0a641-6fcf-4cb1-b302-0aa19633e293 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:35.289476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:35.291504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.292153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.292596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.293738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:35.296766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:35.300905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 973/1919] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:35.319575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:35.324274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:35.348986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 947/1920] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:35 2022] POST /identity/v3/projects => generated 385 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:35.350269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:35.398282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:35.470220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c44219-70ce-4c23-81fc-6233ed2acca7 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:35.486415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283b22-7e59-4f3e-b1d0-dd29f7e0fd4c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:35.493901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:35.494688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.495037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.495564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.496867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:35.500736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:35.511435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:35.553503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 948/1921] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:35 2022] POST /identity/v3/users => generated 299 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:35.554988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:35.570194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efeac8df-cb54-48f1-9cc1-f980b0bc54b6 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:35.580629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:35.581658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.582821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.582821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.583639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:35.583800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:35.584714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.585148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.585590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.600805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:35.617658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 949/1922] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:35 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:35.621991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:35.622767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:35.624058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:35.624387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:35.639968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584537c0-ec93-4bcf-97d6-629dc5d15cdc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:35.649691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:35.650532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.650917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.651316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] PATH_INFO: `/v3/projects/8e970e8e99d041e6bc97a47d4e43fc50` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.653318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e970e8e99d041e6bc97a47d4e43fc50)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:35.654371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 974/1923] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:35.655944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:35.672106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa7b74-8c92-48d7-b531-596fc842a6f6 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:35.672329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:35.679428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:35.680473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.681642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.682143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] PATH_INFO: `/v3/projects/626ac4730d9445d69b97894791c4026a/users/312aa678623d4ebe9f717d86a007998a/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.723157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] RBAC: Authorizing `identity:create_grant(project_id=626ac4730d9445d69b97894791c4026a, user_id=312aa678623d4ebe9f717d86a007998a, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:35.728785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:35.755787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 975/1924] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:35 2022] PUT /identity/v3/projects/626ac4730d9445d69b97894791c4026a/users/312aa678623d4ebe9f717d86a007998a/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:35.762851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:35.792383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e970e8e99d041e6bc97a47d4e43fc50', 'request_id': 'req-a67794fc-529a-4867-a83b-2492032a82fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.804440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e970e8e99d041e6bc97a47d4e43fc50', 'request_id': 'req-a67794fc-529a-4867-a83b-2492032a82fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.814046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e970e8e99d041e6bc97a47d4e43fc50', 'request_id': 'req-a67794fc-529a-4867-a83b-2492032a82fd'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.854041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] The token cache is being invalidate because project 8e970e8e99d041e6bc97a47d4e43fc50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:35.854332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.855220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a67794fc-529a-4867-a83b-2492032a82fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:35.857619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 950/1925] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:35 2022] DELETE /identity/v3/projects/8e970e8e99d041e6bc97a47d4e43fc50 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:35.862124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da4170-c54b-4abb-ae3f-d9ed5f37fe9b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.862545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da4170-c54b-4abb-ae3f-d9ed5f37fe9b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.862826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da4170-c54b-4abb-ae3f-d9ed5f37fe9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.885521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c7da4170-c54b-4abb-ae3f-d9ed5f37fe9b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:35.886778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69a14b-894f-4e1c-bc4c-83d499e03836 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:35.887443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c7da4170-c54b-4abb-ae3f-d9ed5f37fe9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:35.971707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:35.972319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.976789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.976789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:35.976789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:35.976789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:35.987657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 951/1926] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:35.994911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8846d2-3bbf-4374-b7e3-306e718108c0 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:35.995471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8846d2-3bbf-4374-b7e3-306e718108c0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:35.995954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8846d2-3bbf-4374-b7e3-306e718108c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.006150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 976/1927] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:35 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:36.021586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.024566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:36.043467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1b8846d2-3bbf-4374-b7e3-306e718108c0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:36.044290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1b8846d2-3bbf-4374-b7e3-306e718108c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:36.056013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:36.061677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43463648-d61c-4262-910d-0b901c09bae9 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.073854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.073854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.073854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.073854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] PATH_INFO: `/v3/projects/626ac4730d9445d69b97894791c4026a/users/312aa678623d4ebe9f717d86a007998a/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.078562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 952/1928] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:36.084257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8846d2-3bbf-4374-b7e3-306e718108c0 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.089660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=626ac4730d9445d69b97894791c4026a, user_id=312aa678623d4ebe9f717d86a007998a, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.102062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.104010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8846d2-3bbf-4374-b7e3-306e718108c0 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.111072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763b042b-2f91-4fbe-84e3-541a3969eb17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.112126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763b042b-2f91-4fbe-84e3-541a3969eb17 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.112494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763b042b-2f91-4fbe-84e3-541a3969eb17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.113293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763b042b-2f91-4fbe-84e3-541a3969eb17 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.114496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763b042b-2f91-4fbe-84e3-541a3969eb17 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:36.115202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763b042b-2f91-4fbe-84e3-541a3969eb17 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.117582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763b042b-2f91-4fbe-84e3-541a3969eb17 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.139036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 977/1929] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:36 2022] PUT /identity/v3/projects/626ac4730d9445d69b97894791c4026a/users/312aa678623d4ebe9f717d86a007998a/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:36.139790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 953/1930] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:36.157197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.161805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba8e626-5a1b-4266-b16a-5721003a0ca3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.162425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba8e626-5a1b-4266-b16a-5721003a0ca3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.163432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba8e626-5a1b-4266-b16a-5721003a0ca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.270354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-bba8e626-5a1b-4266-b16a-5721003a0ca3 None None] MFA Rules not processed for user `312aa678623d4ebe9f717d86a007998a`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:36.272866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bba8e626-5a1b-4266-b16a-5721003a0ca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:36.347641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:INSERT=4,keystone:SELECT=89,keystone:DELETE=7 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:36.369690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6eefb9-b057-4c83-9180-72f675db19bd admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.384932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.386172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.387450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.387894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.388698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.391564 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.455424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 954/1931] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:36 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:36.464244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 978/1932] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:36 2022] POST /identity/v3/projects => generated 367 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:36.468054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba8e626-5a1b-4266-b16a-5721003a0ca3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.485594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba8e626-5a1b-4266-b16a-5721003a0ca3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.493940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.496003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:36.624267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78579731-72bd-4997-b52c-4871e13a1515 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.628608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.629256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.629583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.630008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.632176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.637399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.638181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.640047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.643495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.651704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.706890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 955/1933] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:36 2022] POST /identity/v3/users => generated 284 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:36.712603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.725410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304ec04-f08c-466c-9445-247be8d420f5 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.731280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.733986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.734439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.735481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.736430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.738719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.745597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.747135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.759755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 956/1934] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:36 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:36.760192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.762099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:36.768975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 979/1935] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:36 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:36.769940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.778107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba38866-39e3-4210-9a2c-a1a4565aeda1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.793763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.793763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.793763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.793763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] PATH_INFO: `/v3/users/1d542c14c4b149be8ee26c83697d22f9` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.794624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d542c14c4b149be8ee26c83697d22f9)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.795143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.799030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281762ae-e444-4c55-b150-a46f536601b7 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.804921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.805788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.806077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.806350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.807908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.810388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.828179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 980/1936] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:36 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:36.828999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.842672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ec0e36-e28b-45a9-b878-cf39bbed8ec0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:36.850806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:36.851455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:36.851737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:36.852056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] PATH_INFO: `/v3/users/873ecd30f8984b8ca7a6f58c829b0f4f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:36.853495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] RBAC: Authorizing `identity:delete_user(user_id=873ecd30f8984b8ca7a6f58c829b0f4f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:36.855434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:36.871520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d542c14c4b149be8ee26c83697d22f9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:36.877361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:36.882572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d542c14c4b149be8ee26c83697d22f9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:36.897939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d542c14c4b149be8ee26c83697d22f9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:36.898674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d542c14c4b149be8ee26c83697d22f9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:36.899325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d542c14c4b149be8ee26c83697d22f9'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:36.930570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 957/1937] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:36 2022] DELETE /identity/v3/users/1d542c14c4b149be8ee26c83697d22f9 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:36.935960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:36.938641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:36.956570 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '873ecd30f8984b8ca7a6f58c829b0f4f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:36.978281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:36.986505 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '873ecd30f8984b8ca7a6f58c829b0f4f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:36.987711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '873ecd30f8984b8ca7a6f58c829b0f4f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:37.002406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '873ecd30f8984b8ca7a6f58c829b0f4f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:37.024302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9820ecc1-37db-4bd9-b020-b741d4415681 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '873ecd30f8984b8ca7a6f58c829b0f4f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:37.039438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 981/1938] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:36 2022] DELETE /identity/v3/users/873ecd30f8984b8ca7a6f58c829b0f4f => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:37.046783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15298908-7108-4cc2-8e80-48db5694a810 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.047926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15298908-7108-4cc2-8e80-48db5694a810 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.047926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15298908-7108-4cc2-8e80-48db5694a810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.086324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-15298908-7108-4cc2-8e80-48db5694a810 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:37.088136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-15298908-7108-4cc2-8e80-48db5694a810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.100042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53fe6c5-1d28-4647-9c99-b1446469ddf0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:37.188264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 982/1939] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:37.190197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187dba8-aa80-4f82-8d33-a11a04601027 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.190447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187dba8-aa80-4f82-8d33-a11a04601027 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.190716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187dba8-aa80-4f82-8d33-a11a04601027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.210172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:37.211061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.211611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.212024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] PATH_INFO: `/v3/projects/4e9d243adc904c0cafc91c7062c357ac/users/8b8525474a7946f09a160cd849d3175c/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.221603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b187dba8-aa80-4f82-8d33-a11a04601027 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:37.224464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b187dba8-aa80-4f82-8d33-a11a04601027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.230721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e9d243adc904c0cafc91c7062c357ac, user_id=8b8525474a7946f09a160cd849d3175c, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:37.233678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:37.233797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 983/1940] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:37.235190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.235565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.236040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.254947 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 958/1941] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:36 2022] PUT /identity/v3/projects/4e9d243adc904c0cafc91c7062c357ac/users/8b8525474a7946f09a160cd849d3175c/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:37.256049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:37.278804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.310824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] MFA Rules not processed for user `0d0911420819469e83dc393a41f1a66f`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:37.312762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.378203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d16202a-d271-4c84-b42c-a721a5867fb3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:37.383434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89339b0-88f6-40db-8f27-b88484f59cc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:37.384333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89339b0-88f6-40db-8f27-b88484f59cc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.384724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89339b0-88f6-40db-8f27-b88484f59cc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.385091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89339b0-88f6-40db-8f27-b88484f59cc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.404979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 984/1942] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:37.409565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:37.411219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.430509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5e719b-d969-46f8-a070-e60e9f577568 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:37.436297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:37.437070 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.437398 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.437695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.455263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:37.474021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89339b0-88f6-40db-8f27-b88484f59cc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:37.476679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89339b0-88f6-40db-8f27-b88484f59cc0 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:37.484053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:37.494052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 959/1943] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:37.496100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.496557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.496892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.541473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] MFA Rules not processed for user `8b8525474a7946f09a160cd849d3175c`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:37.542461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.574462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:37.577636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:37.593223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 985/1944] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:37.594339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:37.595942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.621031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc299d-75bf-4e31-a9fe-4a289775aa51 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:37.628651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:37.630691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.630691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.630691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.631825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:37.635825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:37.675682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 960/1945] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:37.686965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:37.690812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:37.718623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1731964-465a-4647-93b5-d0229fde62bf None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:37.731244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:37.732574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.733271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.733770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.734776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:37.738765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:37.746965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 986/1946] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/projects => generated 373 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:37.748958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:37.820805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 961/1947] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/projects => generated 367 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:37.827978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:37.911576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1ef6b0-295a-43af-bc51-3d85f38e7631 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:37.946726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a071d32f-d541-4654-855b-d82ba9af9be0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:37.954828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:37.955376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:37.955827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:37.956265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:37.957492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:37.961679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.019856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.021331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.022837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.022837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.023908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.027496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 962/1948] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/users => generated 290 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:38.032786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.032966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.049151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9175cc-4b0b-44f3-b918-d6d45be4783b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.060817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.064641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.064932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.065230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.066500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.068580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.080657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 987/1949] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:37 2022] POST /identity/v3/users => generated 293 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:38.082935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.097483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 963/1950] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:38 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:38.102970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.114634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29a6f62-210e-478f-be7c-daccd349f832 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.124489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.125101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c198a-cfca-44c7-8c3e-e8a37ff206ce admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.126079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.126676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.127268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.128864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.131215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.131900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.132238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.132906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] PATH_INFO: `/v3/projects/2dc7f683f3544000bb789c4f540414af/users/06a07deca12841309bf2160d56172fcf/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.133015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.155618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dc7f683f3544000bb789c4f540414af, user_id=06a07deca12841309bf2160d56172fcf, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.160532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 988/1951] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:38 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:38.162362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.162490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.185944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8760a3ef-2085-4f3b-83ee-b8f03481811a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.200219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.201750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.202373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.202927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] PATH_INFO: `/v3/projects/e4177fdf3e6e41afabcc047fe5caf7ad/users/447a8c4337924918b0572a405f6ad868/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.225562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:38.247430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 964/1952] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:38 2022] PUT /identity/v3/projects/2dc7f683f3544000bb789c4f540414af/users/06a07deca12841309bf2160d56172fcf/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:38.248852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.285948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4177fdf3e6e41afabcc047fe5caf7ad, user_id=447a8c4337924918b0572a405f6ad868, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.291525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.293562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:38.318128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 989/1953] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:38 2022] PUT /identity/v3/projects/e4177fdf3e6e41afabcc047fe5caf7ad/users/447a8c4337924918b0572a405f6ad868/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:38.324298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.327636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:38.397187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87505d01-358f-4907-9025-a8374812b176 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.452997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d873ac2-b1de-4e83-ad28-58ad8a1b1cc7 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.459002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.459764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.460035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.460304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] PATH_INFO: `/v3/projects/616d8c6feff34df68117b541eb6b6ecd` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.461565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] RBAC: Authorizing `identity:delete_project(project_id=616d8c6feff34df68117b541eb6b6ecd)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.463551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.468001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.468615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.468889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.469183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.469867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.471450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.487820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 965/1954] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:38 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:38.490666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.493434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:38.508201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c20ad6-f83e-47df-ac9a-e7449fc313cc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.513636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.514277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.514549 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.514813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] PATH_INFO: `/v3/projects/f1683bdac0ee4862861118a5a4bd9996` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.515949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1683bdac0ee4862861118a5a4bd9996)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.517992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.532163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '616d8c6feff34df68117b541eb6b6ecd', 'request_id': 'req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.542581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '616d8c6feff34df68117b541eb6b6ecd', 'request_id': 'req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.554496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '616d8c6feff34df68117b541eb6b6ecd', 'request_id': 'req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.571951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] The token cache is being invalidate because project 616d8c6feff34df68117b541eb6b6ecd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:38.572772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.574092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.575824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 990/1955] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:38 2022] DELETE /identity/v3/projects/616d8c6feff34df68117b541eb6b6ecd => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:38.576622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.578453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:38.582701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1683bdac0ee4862861118a5a4bd9996', 'request_id': 'req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.594127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1683bdac0ee4862861118a5a4bd9996', 'request_id': 'req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.603217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1683bdac0ee4862861118a5a4bd9996', 'request_id': 'req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.615559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] The token cache is being invalidate because project f1683bdac0ee4862861118a5a4bd9996 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:38.615812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.616560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.618242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 966/1956] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:38 2022] DELETE /identity/v3/projects/f1683bdac0ee4862861118a5a4bd9996 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:38.618997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.620560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:38.670419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8b2e2d-3ee5-4548-b8c4-99fc8aed8412 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.681746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:38.698227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:38.704837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f34edd-27e8-47a1-8b8b-583b1d4b3452 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.713362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.714199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.714597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.715176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] PATH_INFO: `/v3/projects/2dc7f683f3544000bb789c4f540414af/users/06a07deca12841309bf2160d56172fcf/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.718612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dc7f683f3544000bb789c4f540414af, user_id=06a07deca12841309bf2160d56172fcf, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.722062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.735243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 967/1957] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:38 2022] PUT /identity/v3/projects/2dc7f683f3544000bb789c4f540414af/users/06a07deca12841309bf2160d56172fcf/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:38.735983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.737711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:38.761918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.762467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.762892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.763453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.764056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.766236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.784080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 991/1958] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:38 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:38.785094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:38.787180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:38.801514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176d2c88-a53d-48f1-b3be-fef82a248a72 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.843306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b495c396-2880-48a2-a7e0-e8b0c8489373 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:38.849610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.850361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.850720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.851120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] PATH_INFO: `/v3/users/c51424434ab84373b959a1b14cf46091` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.852509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c51424434ab84373b959a1b14cf46091)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.854513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.896803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:38.898052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:38.898615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:38.901366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] PATH_INFO: `/v3/users/312aa678623d4ebe9f717d86a007998a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:38.901844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=312aa678623d4ebe9f717d86a007998a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:38.905960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:38.939412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c51424434ab84373b959a1b14cf46091'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.952374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c51424434ab84373b959a1b14cf46091'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.970885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c51424434ab84373b959a1b14cf46091'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.971496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c51424434ab84373b959a1b14cf46091'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:38.977780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-14543c0b-7441-46fc-8661-b64d748dc14d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c51424434ab84373b959a1b14cf46091'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.004215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '312aa678623d4ebe9f717d86a007998a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.009349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 968/1959] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:38 2022] DELETE /identity/v3/users/c51424434ab84373b959a1b14cf46091 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:39.009349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:39.009349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:39.009349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:39.036795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '312aa678623d4ebe9f717d86a007998a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.037711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '312aa678623d4ebe9f717d86a007998a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.047559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '312aa678623d4ebe9f717d86a007998a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.054423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] MFA Rules not processed for user `06a07deca12841309bf2160d56172fcf`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:39.055788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:39.057886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '312aa678623d4ebe9f717d86a007998a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.065349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 992/1960] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:38 2022] DELETE /identity/v3/users/312aa678623d4ebe9f717d86a007998a => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:39.066386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:39.068242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:39.166831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:39.191049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:39.223769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 969/1961] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:39 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:39.264772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:39.267604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:39.272319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e385f0e-fe59-448c-b9c5-4b99f254edc2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:39.279877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:39.281147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:39.281524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:39.282008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] PATH_INFO: `/v3/projects/e4177fdf3e6e41afabcc047fe5caf7ad/users/447a8c4337924918b0572a405f6ad868/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:39.285137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4177fdf3e6e41afabcc047fe5caf7ad, user_id=447a8c4337924918b0572a405f6ad868, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:39.288881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:39.303308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 993/1962] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:39 2022] PUT /identity/v3/projects/e4177fdf3e6e41afabcc047fe5caf7ad/users/447a8c4337924918b0572a405f6ad868/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:39.375199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d3dde-b5c6-4532-8641-c53525779d58 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:39.473303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:39.474054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:39.474344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:39.474606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:39.542701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:39.544778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:39.597026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:39.605325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:39.632535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 970/1963] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:39 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:39.633840 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:39.636172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:39.722153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd121a0-b406-4363-ac41-381fb3d24630 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:39.727092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:39.729993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:39.730767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:39.731088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:39.731387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] PATH_INFO: `/v3/projects/6251924d8732447fb5e5f021836e1ffd` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:39.732741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=6251924d8732447fb5e5f021836e1ffd)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:39.734710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:39.750107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc0b6bc-944c-4b71-bb21-d3f248c21f0d service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:39.755897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:39.756626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:39.757092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:39.757620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] PATH_INFO: `/v3/projects/626ac4730d9445d69b97894791c4026a` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:39.758870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=626ac4730d9445d69b97894791c4026a)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:39.761164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:39.804779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6251924d8732447fb5e5f021836e1ffd', 'request_id': 'req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.814919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6251924d8732447fb5e5f021836e1ffd', 'request_id': 'req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.823880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6251924d8732447fb5e5f021836e1ffd', 'request_id': 'req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.825841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '626ac4730d9445d69b97894791c4026a', 'request_id': 'req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.840115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '626ac4730d9445d69b97894791c4026a', 'request_id': 'req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.847580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] The token cache is being invalidate because project 6251924d8732447fb5e5f021836e1ffd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:39.848054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.848188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '626ac4730d9445d69b97894791c4026a', 'request_id': 'req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.849747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.854145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 994/1964] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:39 2022] DELETE /identity/v3/projects/6251924d8732447fb5e5f021836e1ffd => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:39.855021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:39.856884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:39.865081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] The token cache is being invalidate because project 626ac4730d9445d69b97894791c4026a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:39.865560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.866275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b86fd21f-0fc5-4915-9211-1cf2bc9d96d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:39.868901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 971/1965] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:39 2022] DELETE /identity/v3/projects/626ac4730d9445d69b97894791c4026a => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:39.870566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304eaf1d-0a47-42ac-b576-3b3c62bb6a35 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:39.870857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304eaf1d-0a47-42ac-b576-3b3c62bb6a35 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:39.871257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304eaf1d-0a47-42ac-b576-3b3c62bb6a35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:39.894587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-304eaf1d-0a47-42ac-b576-3b3c62bb6a35 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:39.896655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-304eaf1d-0a47-42ac-b576-3b3c62bb6a35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:39.927206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3498af-feef-49f0-8cc3-dfa41db6b3cb admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:39.950477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:39.988046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 972/1966] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:39.989883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a5561e-bcb7-4edf-82ec-68038fa97890 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:39.991260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a5561e-bcb7-4edf-82ec-68038fa97890 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:39.991936 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a5561e-bcb7-4edf-82ec-68038fa97890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.005931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.006757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.007036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.007457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.009274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.015674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-06a5561e-bcb7-4edf-82ec-68038fa97890 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:40.016887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-06a5561e-bcb7-4edf-82ec-68038fa97890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.032787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 973/1967] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:40.032787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc89b09-3b1b-4495-9277-28cbb77b3cb0 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.032787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc89b09-3b1b-4495-9277-28cbb77b3cb0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.032787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc89b09-3b1b-4495-9277-28cbb77b3cb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.064418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7dc89b09-3b1b-4495-9277-28cbb77b3cb0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:40.065772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7dc89b09-3b1b-4495-9277-28cbb77b3cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.081759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 974/1968] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:40.083296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391e51d-1a5a-456d-b2c7-5449e1b0a1e2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.083758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391e51d-1a5a-456d-b2c7-5449e1b0a1e2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.084001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391e51d-1a5a-456d-b2c7-5449e1b0a1e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.097375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.101198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.109378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4391e51d-1a5a-456d-b2c7-5449e1b0a1e2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:40.110774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4391e51d-1a5a-456d-b2c7-5449e1b0a1e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.111066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 995/1969] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:39 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:40.111938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.120017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 975/1970] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:40.124346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4391e51d-1a5a-456d-b2c7-5449e1b0a1e2 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.125827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d79e0d-81c8-4e9f-b261-0cbc782665cf service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.132035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.132663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.132939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.133249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.133912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:40.134213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.136208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.136592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4391e51d-1a5a-456d-b2c7-5449e1b0a1e2 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.142640 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.143419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.143793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.144171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.144942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:40.145439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.147374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.150483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 996/1971] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:40.155955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.169061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 976/1972] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:40.173071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340ad99c-3cbb-494d-9f13-513b0b988b85 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.173663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.185532 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.186759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.187512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.188064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.189269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.192700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.193893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba838672-918b-4a4b-81b0-8c50880b5ac8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.204230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.204786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.205279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.205664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.206583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.209451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.253229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 997/1973] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/projects => generated 359 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:40.258971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.267117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 977/1974] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/projects => generated 381 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:40.273888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.327233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:40.392670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdde882-1b98-489e-9c5c-ce9345ab5a39 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.404966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74e593b-a924-409e-aa50-a8962d709031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.406288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74e593b-a924-409e-aa50-a8962d709031 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.406577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74e593b-a924-409e-aa50-a8962d709031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.406850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74e593b-a924-409e-aa50-a8962d709031 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.409138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74e593b-a924-409e-aa50-a8962d709031 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.411117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74e593b-a924-409e-aa50-a8962d709031 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.440715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0bbff8-9125-4ebf-8af4-2b7662cc86e6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.448995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 978/1975] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/users => generated 291 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:40.450412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c1265-995d-4f15-9142-c366baf781ce None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.450911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c1265-995d-4f15-9142-c366baf781ce None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.451173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c1265-995d-4f15-9142-c366baf781ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.456792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.458095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.458670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.459055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.460702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.463866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.489398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-146c1265-995d-4f15-9142-c366baf781ce None None] MFA Rules not processed for user `447a8c4337924918b0572a405f6ad868`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:40.491200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-146c1265-995d-4f15-9142-c366baf781ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.500732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 998/1976] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/users => generated 280 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:40.501920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.523873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177c8fcf-e8b3-459d-b03b-b23259660927 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.538951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.540173 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.540631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.541106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.541818 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.546181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.561075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 999/1977] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:40.562084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.574722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db790b-1e18-47c5-9da9-d8215ec87f24 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.580679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.583820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.584234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.584724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.586050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.588753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.604061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1000/1978] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:40.605797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.615500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:40.624375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e657ce25-2893-46c4-bdac-f15cfe49bf0d admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.636513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.637323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.637613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.637886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.638563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.640191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.658875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1001/1979] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:40.659849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.673448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf344517-a4ce-428a-9d99-e70ceabec1b0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.679683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.680451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.680825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.681222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.682270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.683801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.697197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1002/1980] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:40.697197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.704595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 979/1981] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:40.705544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146c1265-995d-4f15-9142-c366baf781ce None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.728096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146c1265-995d-4f15-9142-c366baf781ce None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.733585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49f4e2e-60d8-4fb6-af0c-9ec46366b1ab admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.736234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32919db7-2e2b-489b-b39e-3b29af0b7aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.736963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32919db7-2e2b-489b-b39e-3b29af0b7aa5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.737289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32919db7-2e2b-489b-b39e-3b29af0b7aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.745970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32919db7-2e2b-489b-b39e-3b29af0b7aa5 admin admin] PATH_INFO: `/v3/projects/93e8c5027d374359aa0e718089c72095/users/56bb368a12ab4e23bb20ebfceb9deaf4/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.746894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.747722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.748117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.748486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] PATH_INFO: `/v3/projects/3fb175b1fccb4ddcb3c663f2a7a23721/users/5aababa552e049abbf89e49ab0e59fcf/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.779422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fb175b1fccb4ddcb3c663f2a7a23721, user_id=5aababa552e049abbf89e49ab0e59fcf, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.783026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.784049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32919db7-2e2b-489b-b39e-3b29af0b7aa5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93e8c5027d374359aa0e718089c72095, user_id=56bb368a12ab4e23bb20ebfceb9deaf4, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:40.786904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32919db7-2e2b-489b-b39e-3b29af0b7aa5 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:40.801519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1003/1982] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:40 2022] PUT /identity/v3/projects/3fb175b1fccb4ddcb3c663f2a7a23721/users/5aababa552e049abbf89e49ab0e59fcf/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:40.802552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:40.805668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 980/1983] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:40 2022] PUT /identity/v3/projects/93e8c5027d374359aa0e718089c72095/users/56bb368a12ab4e23bb20ebfceb9deaf4/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:40.807788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.816899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae0a8cc-69a9-4a06-9421-a783fd8efba5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.818329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae0a8cc-69a9-4a06-9421-a783fd8efba5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.818605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae0a8cc-69a9-4a06-9421-a783fd8efba5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.861833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2ae0a8cc-69a9-4a06-9421-a783fd8efba5 None None] MFA Rules not processed for user `5aababa552e049abbf89e49ab0e59fcf`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:40.865519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2ae0a8cc-69a9-4a06-9421-a783fd8efba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.898474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1326af-4baa-405f-9d69-37d9f6e08add admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:40.953394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 981/1984] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:40.954847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.955130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.955399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.969310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:40.970362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:40.971172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:40.972035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:40.972318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:40.974416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:40.988365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] MFA Rules not processed for user `56bb368a12ab4e23bb20ebfceb9deaf4`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:40.989827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:41.042834 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.044704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.055367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1004/1985] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:41.056065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.064386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 982/1986] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:40 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:41.096053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:41.104551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.106816 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:41.121439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ab171f-c548-4fb3-b5b4-12071789ec8b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.125160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.125976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.126334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.127100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.131731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3cff59-4a5b-46e4-92f5-96f6372806b0 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.136147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.137293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.138037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.138175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.138268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.138533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.139170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.141079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.149191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 983/1987] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:41 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:41.166181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.182323 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a09f96-11a7-4903-b366-b62173d25a93 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.186067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1005/1988] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:41 2022] POST /identity/v3/projects => generated 381 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:41.188125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.189059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.189290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.189697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] PATH_INFO: `/v3/users/56bb368a12ab4e23bb20ebfceb9deaf4` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.189965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.191154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=56bb368a12ab4e23bb20ebfceb9deaf4)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.193560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.252657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56bb368a12ab4e23bb20ebfceb9deaf4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:41.263048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56bb368a12ab4e23bb20ebfceb9deaf4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:41.267506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523c279d-1b75-487b-af16-aa853824c353 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.272582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56bb368a12ab4e23bb20ebfceb9deaf4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:41.273312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56bb368a12ab4e23bb20ebfceb9deaf4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:41.273991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-374331fe-8b56-49ff-acdf-fd0fdb0a3a90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56bb368a12ab4e23bb20ebfceb9deaf4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:41.274864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.275628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.275995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.276667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.277983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.280899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.297668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 984/1989] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:41 2022] DELETE /identity/v3/users/56bb368a12ab4e23bb20ebfceb9deaf4 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:41.302944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.303501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.303916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.305249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1006/1990] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:41 2022] POST /identity/v3/users => generated 297 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:41.309368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.311222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:41.326530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f3fef-f4d4-4f5e-b95b-884c22cd278e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.326746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:41.327782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:41.419503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.421612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.422380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.422988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.423841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.425708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.429804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:41.447466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 985/1991] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:41.457604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1007/1992] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:41 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:41.459461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.462711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:41.466166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.477844 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cd8529-7357-42a9-8680-ebcc604d965c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.486444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.487135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.487419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.487716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] PATH_INFO: `/v3/projects/be3b8cedaa434de5a84de01064248bec/users/9d636bd60c514a88b25a3f893c755b44/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.505798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be3b8cedaa434de5a84de01064248bec, user_id=9d636bd60c514a88b25a3f893c755b44, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.508584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.524020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1008/1993] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:41 2022] PUT /identity/v3/projects/be3b8cedaa434de5a84de01064248bec/users/9d636bd60c514a88b25a3f893c755b44/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:41.528411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.542749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6866e16-1ba2-4498-9e82-68615129ec02 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.553759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:41.609434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.609434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.609434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.609731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.617470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c10809b-01ee-4b23-bdd3-0a0e9f3cb6e8 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.621235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.622824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.623146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8129ca59-9b7c-4bac-a44b-e109abab03ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.624210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8129ca59-9b7c-4bac-a44b-e109abab03ce admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.624639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8129ca59-9b7c-4bac-a44b-e109abab03ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.625058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8129ca59-9b7c-4bac-a44b-e109abab03ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.626003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8129ca59-9b7c-4bac-a44b-e109abab03ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.627898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8129ca59-9b7c-4bac-a44b-e109abab03ce admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.641334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 986/1994] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:23:41.643135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1009/1995] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:41 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:41.648175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.668615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa0c267-1279-4b16-9651-a4ffaaa64296 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.674828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.678113 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.678477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.678825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] PATH_INFO: `/v3/projects/be3b8cedaa434de5a84de01064248bec/users/9d636bd60c514a88b25a3f893c755b44/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.681759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=be3b8cedaa434de5a84de01064248bec, user_id=9d636bd60c514a88b25a3f893c755b44, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.685808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.701914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 987/1996] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:41 2022] PUT /identity/v3/projects/be3b8cedaa434de5a84de01064248bec/users/9d636bd60c514a88b25a3f893c755b44/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:41.720884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.721367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.721728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.755582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] MFA Rules not processed for user `9d636bd60c514a88b25a3f893c755b44`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:41.756959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:41.837679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1010/1997] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:41 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:41.854717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.856704 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:41.933554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a83c154-eadb-4fa9-80c8-4c735a8a9f9a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:41.937952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a6a0d1-514f-4bfe-a3ac-e14b499040b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:41.938705 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a6a0d1-514f-4bfe-a3ac-e14b499040b8 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:41.938975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a6a0d1-514f-4bfe-a3ac-e14b499040b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:41.939687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a6a0d1-514f-4bfe-a3ac-e14b499040b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:41.955599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a6a0d1-514f-4bfe-a3ac-e14b499040b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:41.958054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a6a0d1-514f-4bfe-a3ac-e14b499040b8 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:41.976533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 988/1998] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:41 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:41.980511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:41.982052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:42.140858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb50c8e-f98b-4084-98dd-095457b1b956 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.150426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.151631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.152298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.152894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] PATH_INFO: `/v3/projects/93e8c5027d374359aa0e718089c72095` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.155316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] RBAC: Authorizing `identity:delete_project(project_id=93e8c5027d374359aa0e718089c72095)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.159186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.232379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:42.242709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93e8c5027d374359aa0e718089c72095', 'request_id': 'req-0b8e266d-ce07-4742-8a89-9faab45f532f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:42.250364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93e8c5027d374359aa0e718089c72095', 'request_id': 'req-0b8e266d-ce07-4742-8a89-9faab45f532f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:42.258219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93e8c5027d374359aa0e718089c72095', 'request_id': 'req-0b8e266d-ce07-4742-8a89-9faab45f532f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:42.277464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] The token cache is being invalidate because project 93e8c5027d374359aa0e718089c72095 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:42.277765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:42.279211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-0b8e266d-ce07-4742-8a89-9faab45f532f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:42.281159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1011/1999] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:41 2022] DELETE /identity/v3/projects/93e8c5027d374359aa0e718089c72095 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:42.289182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.289442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.289755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.340511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:42.341888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:42.437464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:42.478153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 989/2000] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:42.493957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf6e8c-c3fc-4dbb-bad9-e2fe65624b75 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.494817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf6e8c-c3fc-4dbb-bad9-e2fe65624b75 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.495524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bf6e8c-c3fc-4dbb-bad9-e2fe65624b75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.529962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-34bf6e8c-c3fc-4dbb-bad9-e2fe65624b75 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:42.531467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-34bf6e8c-c3fc-4dbb-bad9-e2fe65624b75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:42.534948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.536372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:42.542306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1012/2001] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:42.547206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bf6e8c-c3fc-4dbb-bad9-e2fe65624b75 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.558378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bf6e8c-c3fc-4dbb-bad9-e2fe65624b75 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.564067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.565047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.565476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.565842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.566502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:42.566855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.568722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.581018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1013/2002] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:23:42.585806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.597428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee2086f-8533-4169-8443-6833f0337a74 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.607672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.608286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.608556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.608829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.609480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.611441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.616036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd985a88-37ab-46c7-8806-bf0f18225889 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.621302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9d636bd60c514a88b25a3f893c755b44', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be3b8cedaa434de5a84de01064248bec', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.622014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.622358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.622696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.623457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:42.623834 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.626126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.640672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 990/2003] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:23:42.645995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.652126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1014/2004] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:42 2022] POST /identity/v3/projects => generated 369 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:42.656634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.666739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e08378a-d381-4497-a69e-5f71db6c3cdb tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.745947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c558899e-9c22-4497-80ef-8582db0e66e6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.751573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.752390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.753365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.753887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.754773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.757559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.784399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9d636bd60c514a88b25a3f893c755b44', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be3b8cedaa434de5a84de01064248bec', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.785023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.785335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.787062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.787062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.788503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.804486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1015/2005] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:42 2022] POST /identity/v3/users => generated 285 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:42.807508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.822185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb5aca-24e3-4fce-af62-011c7d8f89a5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.827710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.828356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.828698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.829013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.829747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.831427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.841101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 991/2006] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:42 2022] POST /identity/v3/projects => generated 354 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:42.843712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1016/2007] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:42 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:42.846242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.847333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.862993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e3947-9526-4822-8935-b3d23f08bc3a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.929065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01a5cfb-01bd-4908-9b74-19451338c182 tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.933348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9d636bd60c514a88b25a3f893c755b44', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be3b8cedaa434de5a84de01064248bec', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.933914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.934474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.934474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.935319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:42.935625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.937313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.937776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.938428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.938897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.939011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:42.939686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:42.941355 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:42.944582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:42.947539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:42.953403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 992/2008] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:42.960061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1017/2009] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:42 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:42.964168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:42.976332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd5afe1-1023-4560-8afc-71c4227643ce tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:42.982504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5fcd24-c67a-401c-b23e-d6ea9572734b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:42.983521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5fcd24-c67a-401c-b23e-d6ea9572734b admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:42.984043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5fcd24-c67a-401c-b23e-d6ea9572734b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:42.984407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5fcd24-c67a-401c-b23e-d6ea9572734b admin admin] PATH_INFO: `/v3/projects/4f1fe8cae02a410b91854d368a4b959e/users/420d058b274d4c0395163389b3a02aa8/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:43.001880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:43.004142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5fcd24-c67a-401c-b23e-d6ea9572734b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f1fe8cae02a410b91854d368a4b959e, user_id=420d058b274d4c0395163389b3a02aa8, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:43.007264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5fcd24-c67a-401c-b23e-d6ea9572734b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:43.017919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2c33db-b5ce-4904-ba0b-efd7b1a308fa admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:43.028198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 993/2010] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:42 2022] PUT /identity/v3/projects/4f1fe8cae02a410b91854d368a4b959e/users/420d058b274d4c0395163389b3a02aa8/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:43.039520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:43.039810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:43.040079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:43.078473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] MFA Rules not processed for user `420d058b274d4c0395163389b3a02aa8`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:43.079750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:43.086959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9d636bd60c514a88b25a3f893c755b44', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be3b8cedaa434de5a84de01064248bec', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:43.087662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:43.087920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:43.088421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] PATH_INFO: `/v3/projects/f63afd18a2954e03adf2147e5c99db61` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:43.089964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorizing `identity:delete_project(project_id=f63afd18a2954e03adf2147e5c99db61)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:43.092902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:43.158464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f63afd18a2954e03adf2147e5c99db61', 'request_id': 'req-b6232621-e756-4c87-aa65-12b0eca4a70d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.161479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 994/2011] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:43 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:43.173412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f63afd18a2954e03adf2147e5c99db61', 'request_id': 'req-b6232621-e756-4c87-aa65-12b0eca4a70d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.176468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:43.178470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:43.184241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f63afd18a2954e03adf2147e5c99db61', 'request_id': 'req-b6232621-e756-4c87-aa65-12b0eca4a70d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.201204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] The token cache is being invalidate because project f63afd18a2954e03adf2147e5c99db61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:43.201529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.202165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.203768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1018/2012] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:43 2022] DELETE /identity/v3/projects/f63afd18a2954e03adf2147e5c99db61 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:43.288621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a8efc4-6536-47af-9caf-e26197683a07 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:43.362636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe19a5ea-aa89-463d-be4d-5c058f969500 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:43.363272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe19a5ea-aa89-463d-be4d-5c058f969500 service glance] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:43.363522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe19a5ea-aa89-463d-be4d-5c058f969500 service glance] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:43.363941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe19a5ea-aa89-463d-be4d-5c058f969500 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:43.366204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fe19a5ea-aa89-463d-be4d-5c058f969500 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:43.398864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:43.480826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe19a5ea-aa89-463d-be4d-5c058f969500 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:43.482794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe19a5ea-aa89-463d-be4d-5c058f969500 service glance] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:43.485482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:43.487091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:43.506231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 995/2013] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:43 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:43.611467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6232621-e756-4c87-aa65-12b0eca4a70d tempest-VolumeQuotaClassesTest-1242027092 tempest-VolumeQuotaClassesTest-1242027092-project-admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:43.616911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:43.617600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:43.617879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:43.618148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] PATH_INFO: `/v3/users/5aababa552e049abbf89e49ab0e59fcf` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:43.619494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] RBAC: Authorizing `identity:delete_user(user_id=5aababa552e049abbf89e49ab0e59fcf)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:43.621395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:43.674450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:43.701877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5aababa552e049abbf89e49ab0e59fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.722034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5aababa552e049abbf89e49ab0e59fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.722909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5aababa552e049abbf89e49ab0e59fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.734329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5aababa552e049abbf89e49ab0e59fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.742383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5aababa552e049abbf89e49ab0e59fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:43.748205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1019/2014] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:43 2022] DELETE /identity/v3/users/5aababa552e049abbf89e49ab0e59fcf => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:43.753792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:43.754200 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:43.754596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:43.793550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:43.795240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:43.859643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 996/2015] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:43.872525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:43.874562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:43.945514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7be082-ea66-49e1-8ff4-45f8e5c74bfa admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:43.951149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:43.952234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:43.952776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:43.953282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:43.967372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:43.969043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:43.989395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1020/2016] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:44.387256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:44.390626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:44.431492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8f36ba-e78f-4d8c-aba0-0c3eef2362a2 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:44.438654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:44.439432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:44.439851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:44.440332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] PATH_INFO: `/v3/projects/3fb175b1fccb4ddcb3c663f2a7a23721` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:44.444085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fb175b1fccb4ddcb3c663f2a7a23721)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:44.448114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:44.531635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fb175b1fccb4ddcb3c663f2a7a23721', 'request_id': 'req-c896f10a-caec-46b5-8d1d-dcdea1ae257c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.541510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fb175b1fccb4ddcb3c663f2a7a23721', 'request_id': 'req-c896f10a-caec-46b5-8d1d-dcdea1ae257c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.549711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fb175b1fccb4ddcb3c663f2a7a23721', 'request_id': 'req-c896f10a-caec-46b5-8d1d-dcdea1ae257c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.562954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] The token cache is being invalidate because project 3fb175b1fccb4ddcb3c663f2a7a23721 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:44.563184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.563866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.565672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 997/2017] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:44 2022] DELETE /identity/v3/projects/3fb175b1fccb4ddcb3c663f2a7a23721 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:44.575548 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:44.582605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de439ca-0c99-49b8-82a8-59e279c9a2ac service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:44.650912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:44.651969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:44.651969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:44.652221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] PATH_INFO: `/v3/users/9d636bd60c514a88b25a3f893c755b44` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:44.653495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d636bd60c514a88b25a3f893c755b44)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:44.662010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:44.742095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d636bd60c514a88b25a3f893c755b44'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.749915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:44.773545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d636bd60c514a88b25a3f893c755b44'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.774304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d636bd60c514a88b25a3f893c755b44'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.788160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d636bd60c514a88b25a3f893c755b44'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.798498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d39e992d-dc04-4699-bf85-063e681d976b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d636bd60c514a88b25a3f893c755b44'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:44.807667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1021/2018] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:44 2022] DELETE /identity/v3/users/9d636bd60c514a88b25a3f893c755b44 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:45.300842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:45.304914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:45.323122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c896f10a-caec-46b5-8d1d-dcdea1ae257c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:45.388736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:45.418926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:45.419571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.419849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.420121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] PATH_INFO: `/v3/projects/be3b8cedaa434de5a84de01064248bec` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.421404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be3b8cedaa434de5a84de01064248bec)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:45.423548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:45.488082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be3b8cedaa434de5a84de01064248bec', 'request_id': 'req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:45.498061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be3b8cedaa434de5a84de01064248bec', 'request_id': 'req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:45.505342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be3b8cedaa434de5a84de01064248bec', 'request_id': 'req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:45.517984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] The token cache is being invalidate because project be3b8cedaa434de5a84de01064248bec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:45.518480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:45.519299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-2da07895-3dfa-4d5b-b880-7e2fb0fe1798 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:45.521501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 998/2019] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:45 2022] DELETE /identity/v3/projects/be3b8cedaa434de5a84de01064248bec => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:45.527052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b1876-dedb-4b07-bc52-94717d5f522a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.527311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b1876-dedb-4b07-bc52-94717d5f522a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.527586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b1876-dedb-4b07-bc52-94717d5f522a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.550125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ad6b1876-dedb-4b07-bc52-94717d5f522a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:45.551826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ad6b1876-dedb-4b07-bc52-94717d5f522a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:45.654954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1022/2020] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:45.679179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.679736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.680243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.704715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:45.706034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:45.715849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 999/2021] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:23:45.724484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b1876-dedb-4b07-bc52-94717d5f522a None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:45.736491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b1876-dedb-4b07-bc52-94717d5f522a None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:45.743276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:45.743948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.744237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.744674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.745537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:23:45.745983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:45.748417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:45.771418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:45.772116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1023/2022] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:23:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:23:45.776781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:45.777647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:45.789038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead2fc9a-c0ed-4f42-80e5-5dbd7bc81c52 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:45.794376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:45.795006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.795277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.795544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.796201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:45.798182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:45.839205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1024/2023] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:45 2022] POST /identity/v3/projects => generated 401 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:45.844598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:45.856700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc2a6e-6005-4e96-a45f-6777a255ec70 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:45.916767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:45.917445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.917925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.918329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.918528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ac9a41-bdf1-436d-8e10-b570dc97bfc6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:45.920956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:45.923307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:45.923916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.924194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.924474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.925131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:45.927147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:45.955437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1025/2024] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:45 2022] POST /identity/v3/users => generated 301 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:23:45.959683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:45.966292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:45.970508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23711e-ac35-49e9-a28e-e35c1cff8e63 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:45.977621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:45.978348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:45.978634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:45.978950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:45.979609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:45.981346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:45.995111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1026/2025] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:45 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:23:46.000158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:46.013021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:46.014872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:46.025288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e698f5a5-a554-41d7-9a2b-544d8f0bd068 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:46.028749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1000/2026] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:45 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:23:46.034367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce5f810-acf1-451d-b304-110587debcbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:46.035238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce5f810-acf1-451d-b304-110587debcbf admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:46.035615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce5f810-acf1-451d-b304-110587debcbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:46.036029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce5f810-acf1-451d-b304-110587debcbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:46.037066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce5f810-acf1-451d-b304-110587debcbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:46.039003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce5f810-acf1-451d-b304-110587debcbf admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:46.051341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1027/2027] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:45 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:46.055854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:46.072397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f0c19d-8960-4112-98ee-d0a62161d0b4 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:46.081162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:46.082237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:46.082638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:46.083050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] PATH_INFO: `/v3/projects/a87cccce95e74cd7a833c27560332386/users/8b7f06bc456549f2a819354372b5da72/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:46.104624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=a87cccce95e74cd7a833c27560332386, user_id=8b7f06bc456549f2a819354372b5da72, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:46.107857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:46.129889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1001/2028] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:46 2022] PUT /identity/v3/projects/a87cccce95e74cd7a833c27560332386/users/8b7f06bc456549f2a819354372b5da72/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:46.141184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e713ea-0d1a-41c2-8e80-3aabe1bb8338 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:46.141470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e713ea-0d1a-41c2-8e80-3aabe1bb8338 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:46.141932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e713ea-0d1a-41c2-8e80-3aabe1bb8338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:46.156312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=93,keystone:DELETE=2,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:46.212648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-93e713ea-0d1a-41c2-8e80-3aabe1bb8338 None None] MFA Rules not processed for user `8b7f06bc456549f2a819354372b5da72`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:46.215902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-93e713ea-0d1a-41c2-8e80-3aabe1bb8338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:46.336195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1028/2029] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:46 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:46.340798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:46.463488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b842f-282a-439d-90b8-554cfdb4b3ad admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:46.482658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:46.483404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:46.483805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:46.484086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:46.498668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:46.513453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:46.570167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1002/2030] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:23:46 2022] POST /identity/v3/projects => generated 401 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:46.575065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e713ea-0d1a-41c2-8e80-3aabe1bb8338 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:46.590939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e713ea-0d1a-41c2-8e80-3aabe1bb8338 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:46.687146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:46.687963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:46.688357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:46.688688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:46.691195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:46.702683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:46.746838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1029/2031] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:23:46 2022] POST /identity/v3/users => generated 307 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:23:46.747566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:46.772327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb00109-ed24-4741-868e-2bbccfd7999f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:46.782284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:46.783403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:46.783907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:46.784264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:46.786053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:46.788602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:46.818564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1003/2032] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:46 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:46.832208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:46.861949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65f062f-c6db-4ee4-bf44-8e2323895353 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:46.870878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:46.871952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:46.872394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:46.872853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] PATH_INFO: `/v3/projects/21e291f6e7d74071a553a1827f6bea0c/users/85a2e4d773ba4b52ba97afd6c8552bb2/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:46.930264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21e291f6e7d74071a553a1827f6bea0c, user_id=85a2e4d773ba4b52ba97afd6c8552bb2, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:46.933307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:46.965170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1030/2033] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:46 2022] PUT /identity/v3/projects/21e291f6e7d74071a553a1827f6bea0c/users/85a2e4d773ba4b52ba97afd6c8552bb2/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:46.971069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:46.987742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e328be-8c8f-4189-85f0-d387bd2017c6 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:47.104877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e787ff05-b8cd-4eb6-8d75-8d962a3c8f4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:47.105745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e787ff05-b8cd-4eb6-8d75-8d962a3c8f4a admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:47.106617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e787ff05-b8cd-4eb6-8d75-8d962a3c8f4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:47.107835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e787ff05-b8cd-4eb6-8d75-8d962a3c8f4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:47.109199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e787ff05-b8cd-4eb6-8d75-8d962a3c8f4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:47.112364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e787ff05-b8cd-4eb6-8d75-8d962a3c8f4a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:47.148292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1004/2034] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:23:46 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:23:47.152893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:47.167394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010485f5-5c62-4211-8f49-d668314870e0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:47.173611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:47.174827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:47.175102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:47.175373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] PATH_INFO: `/v3/projects/21e291f6e7d74071a553a1827f6bea0c/users/85a2e4d773ba4b52ba97afd6c8552bb2/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:47.178068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21e291f6e7d74071a553a1827f6bea0c, user_id=85a2e4d773ba4b52ba97afd6c8552bb2, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:47.185815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:47.239331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1031/2035] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:23:47 2022] PUT /identity/v3/projects/21e291f6e7d74071a553a1827f6bea0c/users/85a2e4d773ba4b52ba97afd6c8552bb2/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:23:47.241387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684fbde9-6aca-498a-89f8-a8c184a94ba9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:47.241678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684fbde9-6aca-498a-89f8-a8c184a94ba9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:47.242033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684fbde9-6aca-498a-89f8-a8c184a94ba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:47.282825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:47.288562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-684fbde9-6aca-498a-89f8-a8c184a94ba9 None None] MFA Rules not processed for user `85a2e4d773ba4b52ba97afd6c8552bb2`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:47.291915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-684fbde9-6aca-498a-89f8-a8c184a94ba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:47.392756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1005/2036] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:47 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:47.429524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:47.508814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:47.519314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea01c957-0db2-446a-b65c-80730c0958b5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:47.524607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:47.524774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:47.525041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:47.525434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:47.536263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:47.537864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:47.553511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1032/2037] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:47 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:47.767457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684fbde9-6aca-498a-89f8-a8c184a94ba9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:47.784393 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684fbde9-6aca-498a-89f8-a8c184a94ba9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:47.789272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c9e64-7d91-4d0a-bc7c-feec1d34c612 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:47.790161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c9e64-7d91-4d0a-bc7c-feec1d34c612 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:47.790441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c9e64-7d91-4d0a-bc7c-feec1d34c612 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:47.790708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c9e64-7d91-4d0a-bc7c-feec1d34c612 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:47.877316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c9e64-7d91-4d0a-bc7c-feec1d34c612 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:47.879092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c9e64-7d91-4d0a-bc7c-feec1d34c612 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:47.895122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1006/2038] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:47 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:23:51.022438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:51.024050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:51.164213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027d67ba-82de-4db5-b908-29c6c75cbc7f service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:51.171780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:51.172752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:51.173255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:51.173723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] PATH_INFO: `/v3/users/8b8525474a7946f09a160cd849d3175c` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:51.175114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b8525474a7946f09a160cd849d3175c)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:51.179573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:51.247067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b8525474a7946f09a160cd849d3175c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:51.267969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b8525474a7946f09a160cd849d3175c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:51.270632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b8525474a7946f09a160cd849d3175c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:51.282567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b8525474a7946f09a160cd849d3175c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:51.295990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b8525474a7946f09a160cd849d3175c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:51.323277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1033/2039] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:51 2022] DELETE /identity/v3/users/8b8525474a7946f09a160cd849d3175c => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:51.328671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:51.329340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:51.329340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:51.354339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:23:51.355620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:51.445692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1007/2040] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:23:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:23:51.456313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:51.457870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:51.564396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59aed2a-0a8d-4145-8051-88a3a1d29d82 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:51.569221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:51.570172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:51.570766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:51.571181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:51.582880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:51.584894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:51.596830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1034/2041] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:23:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:23:52.300296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:52.303450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:52.340391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75201dd3-80a8-450a-9772-b802254fbf04 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:52.360304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:52.364234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:52.366830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:52.367306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] PATH_INFO: `/v3/projects/4e9d243adc904c0cafc91c7062c357ac` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:52.369304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e9d243adc904c0cafc91c7062c357ac)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:52.377494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:52.447628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:52.478247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e9d243adc904c0cafc91c7062c357ac', 'request_id': 'req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.493598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e9d243adc904c0cafc91c7062c357ac', 'request_id': 'req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.504728 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e9d243adc904c0cafc91c7062c357ac', 'request_id': 'req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.529559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] The token cache is being invalidate because project 4e9d243adc904c0cafc91c7062c357ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:52.529559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.529559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.529559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1008/2042] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:52 2022] DELETE /identity/v3/projects/4e9d243adc904c0cafc91c7062c357ac => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:52.536384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:52.556423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7724bfe8-ee83-4403-ab3b-534f96323ce3 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:52.571271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:23:52.689924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:52.690894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:52.691334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:52.692088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] PATH_INFO: `/v3/users/06a07deca12841309bf2160d56172fcf` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:52.697511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06a07deca12841309bf2160d56172fcf)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:52.700342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:52.767565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06a07deca12841309bf2160d56172fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.809105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06a07deca12841309bf2160d56172fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.811406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06a07deca12841309bf2160d56172fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.822995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06a07deca12841309bf2160d56172fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.833015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06a07deca12841309bf2160d56172fcf'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:52.846526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1035/2043] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:23:52 2022] DELETE /identity/v3/users/06a07deca12841309bf2160d56172fcf => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:23:53.298821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:23:53.301456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:23:53.329024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b671cb-b7d2-46d7-80ce-0f29368cdc1a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:23:53.415708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:23:53.416662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:23:53.416959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:23:53.417623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] PATH_INFO: `/v3/projects/2dc7f683f3544000bb789c4f540414af` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:23:53.419732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2dc7f683f3544000bb789c4f540414af)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:23:53.422355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:23:53.495016 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2dc7f683f3544000bb789c4f540414af', 'request_id': 'req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:53.506809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2dc7f683f3544000bb789c4f540414af', 'request_id': 'req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:53.528912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2dc7f683f3544000bb789c4f540414af', 'request_id': 'req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:53.550541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] The token cache is being invalidate because project 2dc7f683f3544000bb789c4f540414af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:23:53.552167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:53.554327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8dc7a5e6-89d0-4400-b1b0-e83bbafe5b6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:23:53.556696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1009/2044] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:23:53 2022] DELETE /identity/v3/projects/2dc7f683f3544000bb789c4f540414af => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:01.621150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:01.623284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:01.637381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce9d1-4e1a-43ce-b517-612c9585c3a9 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:01.719400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:01.721106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:01.721106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:01.721400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] PATH_INFO: `/v3/users/8b7f06bc456549f2a819354372b5da72` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:01.723498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b7f06bc456549f2a819354372b5da72)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:01.725659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:01.764162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:01.778894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b7f06bc456549f2a819354372b5da72'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:01.791412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b7f06bc456549f2a819354372b5da72'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:01.792093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b7f06bc456549f2a819354372b5da72'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:01.800751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b7f06bc456549f2a819354372b5da72'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:01.808657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b7f06bc456549f2a819354372b5da72'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:01.815066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1036/2045] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:01 2022] DELETE /identity/v3/users/8b7f06bc456549f2a819354372b5da72 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:01.820013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:01.820778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:01.820931 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:01.842317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:01.843548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:01.893873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:01.914455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1010/2046] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:01.921565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:01.923383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:01.999095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d172efff-0435-4447-b6d4-6c2b5470d3e1 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:02.004985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:02.005671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:02.006394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:02.006394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:02.017892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:02.019618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:02.036645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1037/2047] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:02.351463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:02.355486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:02.356859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:02.358387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:02.367294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd92135-a47c-49ad-898b-6c146801df73 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:02.371412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ef38d0-a2a4-49a9-9a28-350ac61c11e4 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:02.372231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:02.372981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:02.373468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:02.373752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] PATH_INFO: `/v3/projects/a87cccce95e74cd7a833c27560332386` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:02.375073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a87cccce95e74cd7a833c27560332386)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:02.376271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:02.377000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:02.377170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:02.377276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:02.377613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] PATH_INFO: `/v3/users/37bb7d32464940aeaf2a144304a532d3` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:02.378918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] RBAC: Authorizing `identity:delete_user(user_id=37bb7d32464940aeaf2a144304a532d3)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:02.381288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:02.433814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37bb7d32464940aeaf2a144304a532d3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.434353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a87cccce95e74cd7a833c27560332386', 'request_id': 'req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.443279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a87cccce95e74cd7a833c27560332386', 'request_id': 'req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.446810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37bb7d32464940aeaf2a144304a532d3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.447548 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37bb7d32464940aeaf2a144304a532d3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.450314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a87cccce95e74cd7a833c27560332386', 'request_id': 'req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.457944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37bb7d32464940aeaf2a144304a532d3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.462747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] The token cache is being invalidate because project a87cccce95e74cd7a833c27560332386 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:02.463032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.463664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.465212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1011/2048] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:02 2022] DELETE /identity/v3/projects/a87cccce95e74cd7a833c27560332386 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:02.468946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:02.470209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:02.475218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-42404212-0def-4fa9-8b41-5bd81b3b3103 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37bb7d32464940aeaf2a144304a532d3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.484641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1038/2049] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:02 2022] DELETE /identity/v3/users/37bb7d32464940aeaf2a144304a532d3 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:02.486439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:02.486717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:02.487065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:02.511102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:02.512503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:02.534660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:02.561271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ef724f-e0c1-49a8-8b34-e6d5dee8eb7b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:02.579837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1039/2050] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:02.590828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:02.592712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:02.622427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:02.623062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:02.623341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:02.623609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] PATH_INFO: `/v3/users/85a2e4d773ba4b52ba97afd6c8552bb2` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:02.624730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85a2e4d773ba4b52ba97afd6c8552bb2)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:02.626729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:02.651455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:02.651741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ca3aaa-1e7c-42e2-9052-bbcd35a290c8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:02.655887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:02.656621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:02.656900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:02.657237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:02.670879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:02.672294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:02.678632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85a2e4d773ba4b52ba97afd6c8552bb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.682029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1040/2051] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:02.686659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85a2e4d773ba4b52ba97afd6c8552bb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.695430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85a2e4d773ba4b52ba97afd6c8552bb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.696081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85a2e4d773ba4b52ba97afd6c8552bb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.696702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85a2e4d773ba4b52ba97afd6c8552bb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:02.720722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1012/2052] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:02 2022] DELETE /identity/v3/users/85a2e4d773ba4b52ba97afd6c8552bb2 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:03.074963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.076513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:03.092561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6399f44f-9101-41b4-8028-8deaddda91b3 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.149997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.151960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:03.188538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.189284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.190092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.190296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] PATH_INFO: `/v3/projects/e8e963bad9d04f4bb2a65eb42f86e188` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.191484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8e963bad9d04f4bb2a65eb42f86e188)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.195425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.226715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a75bfc-e9d7-42b0-8180-b6d02fdcb0d7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.231897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.232806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.233172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.233521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] PATH_INFO: `/v3/projects/21e291f6e7d74071a553a1827f6bea0c` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.234747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21e291f6e7d74071a553a1827f6bea0c)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.236832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.265073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8e963bad9d04f4bb2a65eb42f86e188', 'request_id': 'req-d07b7c71-b94d-4174-bb1b-458f9e8d9738'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.274342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8e963bad9d04f4bb2a65eb42f86e188', 'request_id': 'req-d07b7c71-b94d-4174-bb1b-458f9e8d9738'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.282455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8e963bad9d04f4bb2a65eb42f86e188', 'request_id': 'req-d07b7c71-b94d-4174-bb1b-458f9e8d9738'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.291062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21e291f6e7d74071a553a1827f6bea0c', 'request_id': 'req-abba4705-7173-46f0-9837-f1a8ec26dea2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.292903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:03.300439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21e291f6e7d74071a553a1827f6bea0c', 'request_id': 'req-abba4705-7173-46f0-9837-f1a8ec26dea2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.301164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] The token cache is being invalidate because project e8e963bad9d04f4bb2a65eb42f86e188 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:03.301948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.304175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.305678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1041/2053] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:03 2022] DELETE /identity/v3/projects/e8e963bad9d04f4bb2a65eb42f86e188 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:03.311706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21e291f6e7d74071a553a1827f6bea0c', 'request_id': 'req-abba4705-7173-46f0-9837-f1a8ec26dea2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.317758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.319232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:03.324756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] The token cache is being invalidate because project 21e291f6e7d74071a553a1827f6bea0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:03.325039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.325968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-abba4705-7173-46f0-9837-f1a8ec26dea2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.327621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1013/2054] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:03 2022] DELETE /identity/v3/projects/21e291f6e7d74071a553a1827f6bea0c => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:03.332249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9022e-8bc0-4a01-a1ab-3ca7b7caa045 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.332610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9022e-8bc0-4a01-a1ab-3ca7b7caa045 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.332951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9022e-8bc0-4a01-a1ab-3ca7b7caa045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.354275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f9f9022e-8bc0-4a01-a1ab-3ca7b7caa045 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:03.355495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f9f9022e-8bc0-4a01-a1ab-3ca7b7caa045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:03.381189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:03.387399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07b7c71-b94d-4174-bb1b-458f9e8d9738 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.420017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1014/2055] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:03.428493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da424f4-61c1-4d6f-b58e-525cd5c95ce4 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.428799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da424f4-61c1-4d6f-b58e-525cd5c95ce4 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.429312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da424f4-61c1-4d6f-b58e-525cd5c95ce4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.443515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.444221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.444497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.444828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] PATH_INFO: `/v3/users/737cb56f94494015ab44fe6dc4496d9a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.446017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] RBAC: Authorizing `identity:delete_user(user_id=737cb56f94494015ab44fe6dc4496d9a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.447923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.450911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-6da424f4-61c1-4d6f-b58e-525cd5c95ce4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:03.452071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6da424f4-61c1-4d6f-b58e-525cd5c95ce4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:03.462117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1015/2056] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:03.466663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da424f4-61c1-4d6f-b58e-525cd5c95ce4 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.478728 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da424f4-61c1-4d6f-b58e-525cd5c95ce4 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.485534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.486297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.486568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.486861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.487561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:03.487802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.489729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.502058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1016/2057] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:24:03.506854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.507862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '737cb56f94494015ab44fe6dc4496d9a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.516385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f147f40-45d3-43cb-af25-6347445bc279 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.521237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '737cb56f94494015ab44fe6dc4496d9a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.522162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '737cb56f94494015ab44fe6dc4496d9a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.522812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.523397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.523675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.523935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.524581 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.526530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.538462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '737cb56f94494015ab44fe6dc4496d9a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.549084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '737cb56f94494015ab44fe6dc4496d9a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:03.557403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1042/2058] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:03 2022] DELETE /identity/v3/users/737cb56f94494015ab44fe6dc4496d9a => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:03.565230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1017/2059] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:03 2022] POST /identity/v3/projects => generated 385 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:03.568209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.570438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:03.600639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:03.647978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc1972-597a-4b81-bd52-b3f54f7ed324 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.728780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.729931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.730210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.730473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.731110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.733224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.761292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1043/2060] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:03 2022] POST /identity/v3/users => generated 293 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:03.765727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.777699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f249823d-63cf-4844-9052-8af33a78fcfb admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.784158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.785723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.786140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.786545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.787366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.789494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.810525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1018/2061] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:03 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:03.814686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.829367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a86cab-854b-4041-9ae2-31fe2c85f84b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.836792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.837518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.837857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.838288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.838931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.841677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.859956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1044/2062] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:03 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:03.864503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.877216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12b2205-b9d6-4d5f-bc12-71b35f0e9296 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.882922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd06e788-d0ec-4e65-a99a-2c9bef4efddf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:03.883595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd06e788-d0ec-4e65-a99a-2c9bef4efddf admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.884020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd06e788-d0ec-4e65-a99a-2c9bef4efddf admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.884299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd06e788-d0ec-4e65-a99a-2c9bef4efddf admin admin] PATH_INFO: `/v3/projects/363ab7ac0b0c4f37a92d6d4fa8359e8f/users/a8d008b266564a31a2f869f081389dc0/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.905761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd06e788-d0ec-4e65-a99a-2c9bef4efddf admin admin] RBAC: Authorizing `identity:create_grant(project_id=363ab7ac0b0c4f37a92d6d4fa8359e8f, user_id=a8d008b266564a31a2f869f081389dc0, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:03.908889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd06e788-d0ec-4e65-a99a-2c9bef4efddf admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:03.919681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:03.921886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:03.929787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1019/2063] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:03 2022] PUT /identity/v3/projects/363ab7ac0b0c4f37a92d6d4fa8359e8f/users/a8d008b266564a31a2f869f081389dc0/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:03.941190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6d7b8-d260-4157-8ad2-17ad15e008f1 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:03.941536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6d7b8-d260-4157-8ad2-17ad15e008f1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:03.941969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6d7b8-d260-4157-8ad2-17ad15e008f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:03.944583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e964cf25-5da3-4e11-a5af-00dbe245ac7f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:03.970516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8cd6d7b8-d260-4157-8ad2-17ad15e008f1 None None] MFA Rules not processed for user `a8d008b266564a31a2f869f081389dc0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:03.971657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8cd6d7b8-d260-4157-8ad2-17ad15e008f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.003083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.003705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.004000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.004300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] PATH_INFO: `/v3/projects/7fd6329902534d67bb14510a4c27bdd1` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.005466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7fd6329902534d67bb14510a4c27bdd1)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.007380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.044050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1020/2064] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:04.049435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6d7b8-d260-4157-8ad2-17ad15e008f1 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.055237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7fd6329902534d67bb14510a4c27bdd1', 'request_id': 'req-12f1afdc-3482-4d2c-83de-1474b9a56bd8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:04.057060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:04.060994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6d7b8-d260-4157-8ad2-17ad15e008f1 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.062155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7fd6329902534d67bb14510a4c27bdd1', 'request_id': 'req-12f1afdc-3482-4d2c-83de-1474b9a56bd8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:04.070475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7fd6329902534d67bb14510a4c27bdd1', 'request_id': 'req-12f1afdc-3482-4d2c-83de-1474b9a56bd8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:04.086973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] The token cache is being invalidate because project 7fd6329902534d67bb14510a4c27bdd1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:04.087254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:04.088257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-12f1afdc-3482-4d2c-83de-1474b9a56bd8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:04.090254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1045/2065] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:03 2022] DELETE /identity/v3/projects/7fd6329902534d67bb14510a4c27bdd1 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:04.095302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde84278-0a23-4fd4-98ba-d4a6eacca7fa None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.095302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde84278-0a23-4fd4-98ba-d4a6eacca7fa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.095632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde84278-0a23-4fd4-98ba-d4a6eacca7fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.111780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:04.121641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-dde84278-0a23-4fd4-98ba-d4a6eacca7fa None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:04.122891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dde84278-0a23-4fd4-98ba-d4a6eacca7fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.128042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.128658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.128930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.129808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.130523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.132405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.165567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1021/2066] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/projects => generated 383 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:04.167315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.168998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.184743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1046/2067] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:04.192693 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb64fc1-f044-4f2e-abb7-0f84f73816df None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.192973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb64fc1-f044-4f2e-abb7-0f84f73816df None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.193278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb64fc1-f044-4f2e-abb7-0f84f73816df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.212792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-dbb64fc1-f044-4f2e-abb7-0f84f73816df None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:04.214053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dbb64fc1-f044-4f2e-abb7-0f84f73816df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.236607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee783265-e192-4a7c-95cc-2b3fe2b86fc6 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.242599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.243210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.243481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.243747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.244399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.246371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.273349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1022/2068] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/users => generated 298 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:04.274416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.279635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1047/2069] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:04.283870 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb64fc1-f044-4f2e-abb7-0f84f73816df None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.284472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d132acea-9fdb-444c-b313-56a7811e55ac admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.289725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.290321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.290607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.290876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.291530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.293152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.293392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb64fc1-f044-4f2e-abb7-0f84f73816df None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.298409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.299006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.299278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.299542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.300181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:04.300604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.302315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.305652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1023/2070] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:04.309623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.318225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1048/2071] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:24:04.319586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc335793-162e-4ff3-ae3f-6e521de88dbb admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.322787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.324956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.325667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.325942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.326230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] PATH_INFO: `/v3/projects/8c571653728b4cb48c108b13f3aa314b/users/6111552258514603954e3e869f8eec0e/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.334089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046afbca-3891-4e0b-a923-d518eccf4ed0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.339906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.340563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.341275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.341417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.342025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.344106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.350012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c571653728b4cb48c108b13f3aa314b, user_id=6111552258514603954e3e869f8eec0e, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.352981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.368520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1024/2072] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:04 2022] PUT /identity/v3/projects/8c571653728b4cb48c108b13f3aa314b/users/6111552258514603954e3e869f8eec0e/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:04.373365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.385443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1049/2073] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/projects => generated 385 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:04.387163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.445921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:04.459143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833ab56-e94e-49ae-b1b8-573d02bd9808 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.465083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.465729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.466007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.466134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc99601-7724-48d1-9730-a8e8e2403ae1 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.466305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.466964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.468629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.470949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.471560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.471846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.472120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.472805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.474753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.481158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1025/2074] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:04.485576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.498852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18b2ccb-88b6-4df1-a0d4-35109bca5932 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.503432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1050/2075] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/users => generated 293 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:04.505734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d109ba29-5e94-4744-9ec2-2d4616b79c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.506672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d109ba29-5e94-4744-9ec2-2d4616b79c9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.506955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d109ba29-5e94-4744-9ec2-2d4616b79c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.507232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d109ba29-5e94-4744-9ec2-2d4616b79c9b admin admin] PATH_INFO: `/v3/projects/8c571653728b4cb48c108b13f3aa314b/users/6111552258514603954e3e869f8eec0e/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.508909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.509461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d109ba29-5e94-4744-9ec2-2d4616b79c9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c571653728b4cb48c108b13f3aa314b, user_id=6111552258514603954e3e869f8eec0e, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.512354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d109ba29-5e94-4744-9ec2-2d4616b79c9b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.519285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e154b4d-09c5-4ca5-8191-69f24359c42e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.525260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1026/2076] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:04 2022] PUT /identity/v3/projects/8c571653728b4cb48c108b13f3aa314b/users/6111552258514603954e3e869f8eec0e/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:04.535687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.535967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.536236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.550798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:04.568328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] MFA Rules not processed for user `6111552258514603954e3e869f8eec0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:04.569607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.587552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.588190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.588464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.588736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.589406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.591014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.601043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1051/2077] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:04.604916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.615634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4803c5a0-66f1-413e-89f2-ee53b9ef5283 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.620459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.621053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.621426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.621702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.622327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.623905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.638957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1052/2078] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:24:04.640166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.648461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1027/2079] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:04.650121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b9705b-932f-4172-9130-4f5e8dd9039f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.655786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0190994a-19b6-4a1f-b32c-d5c89483e080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.656398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.656525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0190994a-19b6-4a1f-b32c-d5c89483e080 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.656703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0190994a-19b6-4a1f-b32c-d5c89483e080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.656973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0190994a-19b6-4a1f-b32c-d5c89483e080 admin admin] PATH_INFO: `/v3/projects/154ca7bc467b40859e50d485044488dd/users/58d8d12047ec41cda4c292ebe673b257/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.657961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.673825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0190994a-19b6-4a1f-b32c-d5c89483e080 admin admin] RBAC: Authorizing `identity:create_grant(project_id=154ca7bc467b40859e50d485044488dd, user_id=58d8d12047ec41cda4c292ebe673b257, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.676987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0190994a-19b6-4a1f-b32c-d5c89483e080 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.689822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1053/2080] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:04 2022] PUT /identity/v3/projects/154ca7bc467b40859e50d485044488dd/users/58d8d12047ec41cda4c292ebe673b257/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:04.702380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f503f5-3e9d-45b2-a019-8dc102d22bf9 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.702664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f503f5-3e9d-45b2-a019-8dc102d22bf9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.702975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f503f5-3e9d-45b2-a019-8dc102d22bf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.729192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4b07a4-f6c9-4273-a7f1-a4d86cd10257 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.732405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-68f503f5-3e9d-45b2-a019-8dc102d22bf9 None None] MFA Rules not processed for user `58d8d12047ec41cda4c292ebe673b257`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:04.733756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-68f503f5-3e9d-45b2-a019-8dc102d22bf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.783328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.783943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.784221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.784491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.787446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:04.801856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1054/2081] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:04.806372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f503f5-3e9d-45b2-a019-8dc102d22bf9 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.835106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:04.849191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.850643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.859855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1028/2082] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:04.868363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f503f5-3e9d-45b2-a019-8dc102d22bf9 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:04.873882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:04.874707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:04.875192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:04.875646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:04.876447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:04.878924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:04.917260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1055/2083] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/projects => generated 385 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:04.921383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:04.926879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:04.934987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dc324c-a844-4ba0-9f33-48c55f07adb7 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.012251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.012789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.013064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.013363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.014016 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.016024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.043028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1029/2084] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:04 2022] POST /identity/v3/users => generated 299 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:05.046842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:05.060156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5cf4ad-97be-48b3-8a37-5dcc6e67e9b7 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.066826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.067559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.067847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.068162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.068752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.070378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.089190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1056/2085] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:24:05.089793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:05.102889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee8ee53-a508-4006-ab76-a59ab68825ec admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.110450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.111723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.112387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.113008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] PATH_INFO: `/v3/projects/86737680dbc34ed4a2a84d5631f8d5e8/users/34b80db931484105b661010a6bafa437/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.134506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] RBAC: Authorizing `identity:create_grant(project_id=86737680dbc34ed4a2a84d5631f8d5e8, user_id=34b80db931484105b661010a6bafa437, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.137572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.159293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:05.159805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1030/2086] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:05 2022] PUT /identity/v3/projects/86737680dbc34ed4a2a84d5631f8d5e8/users/34b80db931484105b661010a6bafa437/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:05.170866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1ba2e0-77bf-4398-97ac-e85325ba4486 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.250405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5939ca87-1a36-4465-ab8c-24315b2a019f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.251036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939ca87-1a36-4465-ab8c-24315b2a019f admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.251317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939ca87-1a36-4465-ab8c-24315b2a019f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.251589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939ca87-1a36-4465-ab8c-24315b2a019f admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.252315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5939ca87-1a36-4465-ab8c-24315b2a019f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.254233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5939ca87-1a36-4465-ab8c-24315b2a019f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.265631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1057/2087] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:05 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:05.270486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:05.281821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432c6c80-5df2-4039-8ad9-fbd0d2613bcf admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.291620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.291979 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.292247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.292513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] PATH_INFO: `/v3/projects/86737680dbc34ed4a2a84d5631f8d5e8/users/34b80db931484105b661010a6bafa437/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.294535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] RBAC: Authorizing `identity:create_grant(project_id=86737680dbc34ed4a2a84d5631f8d5e8, user_id=34b80db931484105b661010a6bafa437, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.297140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.310227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1031/2088] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:05 2022] PUT /identity/v3/projects/86737680dbc34ed4a2a84d5631f8d5e8/users/34b80db931484105b661010a6bafa437/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:05.322024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b073310-8b50-42d2-928b-31a8ea0c1183 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.322263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b073310-8b50-42d2-928b-31a8ea0c1183 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.322519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b073310-8b50-42d2-928b-31a8ea0c1183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.356195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1b073310-8b50-42d2-928b-31a8ea0c1183 None None] MFA Rules not processed for user `34b80db931484105b661010a6bafa437`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:05.357795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1b073310-8b50-42d2-928b-31a8ea0c1183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:05.437718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1058/2089] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:05 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:05.451310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:05.510507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108eb4d5-43d0-4fa3-a9f4-610c538e9432 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.514371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88ab36e-272b-4fea-8693-7f65d526f3d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.515107 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88ab36e-272b-4fea-8693-7f65d526f3d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.515338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88ab36e-272b-4fea-8693-7f65d526f3d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.515660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88ab36e-272b-4fea-8693-7f65d526f3d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.525089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88ab36e-272b-4fea-8693-7f65d526f3d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.526560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88ab36e-272b-4fea-8693-7f65d526f3d1 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.536210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1032/2090] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:05 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:05.602702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b073310-8b50-42d2-928b-31a8ea0c1183 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:05.628229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:05.669547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b073310-8b50-42d2-928b-31a8ea0c1183 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.674568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.675196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.675469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.675736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] PATH_INFO: `/v3/users/58d8d12047ec41cda4c292ebe673b257` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.676882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=58d8d12047ec41cda4c292ebe673b257)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.679004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.731349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58d8d12047ec41cda4c292ebe673b257'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:05.743671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58d8d12047ec41cda4c292ebe673b257'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:05.744385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58d8d12047ec41cda4c292ebe673b257'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:05.754231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58d8d12047ec41cda4c292ebe673b257'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:05.762772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58d8d12047ec41cda4c292ebe673b257'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:05.769037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1059/2091] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:05 2022] DELETE /identity/v3/users/58d8d12047ec41cda4c292ebe673b257 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:05.775062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.775341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.775607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.795015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:05.796319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:05.849862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:05.861467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1033/2092] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:05.872031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:05.873513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:05.932896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3a963-0037-41c7-8dbd-c576d06d7e1b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:05.937234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:05.937807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:05.938145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:05.938375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:05.951321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:05.952835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:05.963048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1060/2093] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:06.265253 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:06.266978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:06.286878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0feff5-5013-4d63-97b3-8e94bb08c32d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:06.295488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:06.295944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:06.296211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:06.296513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] PATH_INFO: `/v3/projects/154ca7bc467b40859e50d485044488dd` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:06.297641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=154ca7bc467b40859e50d485044488dd)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:06.299564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:06.352320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '154ca7bc467b40859e50d485044488dd', 'request_id': 'req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.363516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '154ca7bc467b40859e50d485044488dd', 'request_id': 'req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.374465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '154ca7bc467b40859e50d485044488dd', 'request_id': 'req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.391689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] The token cache is being invalidate because project 154ca7bc467b40859e50d485044488dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:06.391985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.392747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.394827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1034/2094] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:06 2022] DELETE /identity/v3/projects/154ca7bc467b40859e50d485044488dd => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:06.399425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:06.409151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0063be25-1027-424f-b9f5-395ed5ba82ac service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:06.457245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:06.474205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:06.474867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:06.475286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:06.475590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] PATH_INFO: `/v3/users/34b80db931484105b661010a6bafa437` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:06.476749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34b80db931484105b661010a6bafa437)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:06.478695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:06.527667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34b80db931484105b661010a6bafa437'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.540350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34b80db931484105b661010a6bafa437'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.541582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34b80db931484105b661010a6bafa437'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.553745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34b80db931484105b661010a6bafa437'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.561197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d098e8b2-831a-43bd-b613-2a71c3603730 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34b80db931484105b661010a6bafa437'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:06.567561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1061/2095] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:06 2022] DELETE /identity/v3/users/34b80db931484105b661010a6bafa437 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:07.117063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.118653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:07.132967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e79d0c-b4ae-44bd-ae79-3dbc9518ec65 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.198231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.199171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.199439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.199699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] PATH_INFO: `/v3/projects/86737680dbc34ed4a2a84d5631f8d5e8` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.200803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] RBAC: Authorizing `identity:delete_project(project_id=86737680dbc34ed4a2a84d5631f8d5e8)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.202712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:07.256868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86737680dbc34ed4a2a84d5631f8d5e8', 'request_id': 'req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:07.266918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86737680dbc34ed4a2a84d5631f8d5e8', 'request_id': 'req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:07.274321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86737680dbc34ed4a2a84d5631f8d5e8', 'request_id': 'req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:07.286652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] The token cache is being invalidate because project 86737680dbc34ed4a2a84d5631f8d5e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:07.286652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:07.287357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7a70e673-a371-4b8a-b009-7e5fcc9b5d8c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:07.289097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1035/2096] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:07 2022] DELETE /identity/v3/projects/86737680dbc34ed4a2a84d5631f8d5e8 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:07.294191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbad513-2b2e-4eba-bef8-b37fe042b4c8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.294706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbad513-2b2e-4eba-bef8-b37fe042b4c8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.295121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbad513-2b2e-4eba-bef8-b37fe042b4c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.315433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4bbad513-2b2e-4eba-bef8-b37fe042b4c8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:07.316724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4bbad513-2b2e-4eba-bef8-b37fe042b4c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:07.379449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1062/2097] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:07.387277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b9421-d13f-49c2-8c9e-d95e3ee0849b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.387557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b9421-d13f-49c2-8c9e-d95e3ee0849b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.387941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b9421-d13f-49c2-8c9e-d95e3ee0849b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.409416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7b5b9421-d13f-49c2-8c9e-d95e3ee0849b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:07.410625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7b5b9421-d13f-49c2-8c9e-d95e3ee0849b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:07.419112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1036/2098] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:07.424132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbad513-2b2e-4eba-bef8-b37fe042b4c8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.438627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbad513-2b2e-4eba-bef8-b37fe042b4c8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.445362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.446196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.446567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.446835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.447486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:07.447785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.449732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:07.462537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1063/2099] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:24:07.467136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b9421-d13f-49c2-8c9e-d95e3ee0849b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.473190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:07.477482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b9421-d13f-49c2-8c9e-d95e3ee0849b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.484653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.486120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.486120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.486278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.487124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.489925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:07.522969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1037/2100] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:07 2022] POST /identity/v3/projects => generated 407 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:07.527389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.538362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7a02f-5712-4778-827d-0864d2983653 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.588813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:07.600574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.601371 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.601951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.602643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.603301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.605239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:07.630622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1064/2101] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:07 2022] POST /identity/v3/users => generated 304 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:07.632452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.643165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603512c5-7f01-414e-867c-ed9ef107d450 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.649244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.649899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.650234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.650505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.651312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.652949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:07.665239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1038/2102] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:07.669078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.679516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec7d2a-4453-4ed7-a6b0-878979efebb0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.684216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdae9c4-443e-4276-a039-9cf76d1647d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.684827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdae9c4-443e-4276-a039-9cf76d1647d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.685922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdae9c4-443e-4276-a039-9cf76d1647d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.685922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdae9c4-443e-4276-a039-9cf76d1647d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.686226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdae9c4-443e-4276-a039-9cf76d1647d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.687953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdae9c4-443e-4276-a039-9cf76d1647d9 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:07.701290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1065/2103] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:07.705918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.721335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893d5e3a-d408-4079-b1d0-d976887f10cf admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.726304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.726950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.727228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.727501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] PATH_INFO: `/v3/projects/fcaeee6582514025bb9fa0e468128dd4/users/323456a5ac42467bbd4b1d368c87bde3/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.742432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcaeee6582514025bb9fa0e468128dd4, user_id=323456a5ac42467bbd4b1d368c87bde3, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.745427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:07.759679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1039/2104] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:07 2022] PUT /identity/v3/projects/fcaeee6582514025bb9fa0e468128dd4/users/323456a5ac42467bbd4b1d368c87bde3/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:07.772361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2371dec-2dfb-4063-958c-b1dc9d2615f4 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.772831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2371dec-2dfb-4063-958c-b1dc9d2615f4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.773230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2371dec-2dfb-4063-958c-b1dc9d2615f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.814098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-f2371dec-2dfb-4063-958c-b1dc9d2615f4 None None] MFA Rules not processed for user `323456a5ac42467bbd4b1d368c87bde3`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:07.815428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f2371dec-2dfb-4063-958c-b1dc9d2615f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:07.889488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1066/2105] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:07 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:07.894110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:07.959136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d22262-ce0f-462c-977d-7a2c490e8990 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:07.964582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:07.965259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:07.965536 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:07.965806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:07.966457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:07.968354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.006009 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1040/2106] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:07 2022] POST /identity/v3/projects => generated 407 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:08.007545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2371dec-2dfb-4063-958c-b1dc9d2615f4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:08.022329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2371dec-2dfb-4063-958c-b1dc9d2615f4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:08.087410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:08.088032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.088306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.088574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.089272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:08.091189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.123282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1067/2107] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:08 2022] POST /identity/v3/users => generated 310 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:08.125081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:08.136324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbf4b2d-4132-4e94-b245-d437513d478a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:08.143603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:08.144525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.144525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.144971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.146770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:08.148753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.164751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1041/2108] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:08 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:24:08.171794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:08.183000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8260d47-f435-4b1c-9967-e1a3f3f319e6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:08.189580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:08.190300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.190551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.190823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] PATH_INFO: `/v3/projects/30d414797e99499984020d34de351fd6/users/43f7ce5c18574055bb1e700a179e9310/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.207568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30d414797e99499984020d34de351fd6, user_id=43f7ce5c18574055bb1e700a179e9310, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:08.210212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.226664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1068/2109] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:08 2022] PUT /identity/v3/projects/30d414797e99499984020d34de351fd6/users/43f7ce5c18574055bb1e700a179e9310/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:08.230958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:08.241487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad1a91-9c6a-4e2c-b682-2c61ad3d76fd admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:08.301178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5772df-4b4a-475c-b47f-d481831f83ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:08.302677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5772df-4b4a-475c-b47f-d481831f83ff admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.303102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5772df-4b4a-475c-b47f-d481831f83ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.303370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5772df-4b4a-475c-b47f-d481831f83ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.304036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5772df-4b4a-475c-b47f-d481831f83ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:08.305667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5772df-4b4a-475c-b47f-d481831f83ff admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.316223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1042/2110] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:08 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:08.320707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:08.332113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8844832d-120b-4629-9429-b409c9021808 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:08.338509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:08.339188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.339463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.339788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] PATH_INFO: `/v3/projects/30d414797e99499984020d34de351fd6/users/43f7ce5c18574055bb1e700a179e9310/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.341849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30d414797e99499984020d34de351fd6, user_id=43f7ce5c18574055bb1e700a179e9310, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:08.344509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.357698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1069/2111] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:08 2022] PUT /identity/v3/projects/30d414797e99499984020d34de351fd6/users/43f7ce5c18574055bb1e700a179e9310/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:08.368299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273108ce-5c7d-4fba-a751-33df04fbb6f7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.368798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273108ce-5c7d-4fba-a751-33df04fbb6f7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.369300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273108ce-5c7d-4fba-a751-33df04fbb6f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.406210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-273108ce-5c7d-4fba-a751-33df04fbb6f7 None None] MFA Rules not processed for user `43f7ce5c18574055bb1e700a179e9310`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:08.407444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-273108ce-5c7d-4fba-a751-33df04fbb6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:08.415413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:08.490928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1043/2112] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:08 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:08.502115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:08.503861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:08.518519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:08.565740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd3db21-5e66-47fd-bc1c-bf66ff1cb017 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:08.570069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e645bb-eb21-4fff-a094-d268e1315e34 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:08.570683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e645bb-eb21-4fff-a094-d268e1315e34 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.571012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e645bb-eb21-4fff-a094-d268e1315e34 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.571213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e645bb-eb21-4fff-a094-d268e1315e34 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.582110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e645bb-eb21-4fff-a094-d268e1315e34 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:08.583916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e645bb-eb21-4fff-a094-d268e1315e34 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.597546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1070/2113] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:08 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:08.759716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273108ce-5c7d-4fba-a751-33df04fbb6f7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:08.770410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273108ce-5c7d-4fba-a751-33df04fbb6f7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:08.774621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e84570-9f34-4eb1-a846-a81087325d73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:08.775271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e84570-9f34-4eb1-a846-a81087325d73 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.775636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e84570-9f34-4eb1-a846-a81087325d73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.775909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e84570-9f34-4eb1-a846-a81087325d73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.840123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e84570-9f34-4eb1-a846-a81087325d73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:08.841647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e84570-9f34-4eb1-a846-a81087325d73 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:08.851345 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1044/2114] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:08 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:08.875779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac77dd1-1a41-4518-bf76-7e60b5806e4b None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.876127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac77dd1-1a41-4518-bf76-7e60b5806e4b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.876423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac77dd1-1a41-4518-bf76-7e60b5806e4b None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.878142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1071/2115] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:24:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:24:08.917522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933232c4-cd00-428d-aa68-b4e96b40795b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:08.917857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933232c4-cd00-428d-aa68-b4e96b40795b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:08.918181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933232c4-cd00-428d-aa68-b4e96b40795b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:08.940973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-933232c4-cd00-428d-aa68-b4e96b40795b None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:08.942800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-933232c4-cd00-428d-aa68-b4e96b40795b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:09.043071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1045/2116] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:24:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:09.057039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b937cc93-71b3-45ac-953e-65e57272c7d9 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:09.057348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b937cc93-71b3-45ac-953e-65e57272c7d9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:09.057617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b937cc93-71b3-45ac-953e-65e57272c7d9 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:09.058954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1072/2117] 10.222.0.52 () {54 vars in 961 bytes} [Sat Aug 27 02:24:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:24:09.065190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:09.065737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:09.066244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:09.093510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:09.094996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:09.109938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1046/2118] 10.222.0.52 () {58 vars in 1058 bytes} [Sat Aug 27 02:24:09 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:09.121024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b937cc93-71b3-45ac-953e-65e57272c7d9 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:09.136434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b937cc93-71b3-45ac-953e-65e57272c7d9 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:09.147343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:09.148378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:09.148988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:09.149510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:09.163235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:09.165595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:09.177335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1073/2119] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:09.545949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:09.547999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:09.599199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:09.635021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad26d33-ed09-4414-94df-9a20e20917b6 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:09.640827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:09.641416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:09.641690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:09.641955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] PATH_INFO: `/v3/users/0d0911420819469e83dc393a41f1a66f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:09.643183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d0911420819469e83dc393a41f1a66f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:09.645825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:09.697735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d0911420819469e83dc393a41f1a66f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:09.705336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d0911420819469e83dc393a41f1a66f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:09.714546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d0911420819469e83dc393a41f1a66f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:09.715333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d0911420819469e83dc393a41f1a66f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:09.716012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d0911420819469e83dc393a41f1a66f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:09.737234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1047/2120] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:09 2022] DELETE /identity/v3/users/0d0911420819469e83dc393a41f1a66f => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:09.935119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:09.947150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f786c72f-7896-414a-8db1-8e3500d07361 service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:10.019874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:10.020517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:10.020709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:10.020976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] PATH_INFO: `/v3/projects/46f05e0ab37c4e52afb82496e4554101` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:10.022162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=46f05e0ab37c4e52afb82496e4554101)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:10.024102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:10.077027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46f05e0ab37c4e52afb82496e4554101', 'request_id': 'req-ea040ea3-a011-41b1-86d5-f0b40295e1b6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.083404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46f05e0ab37c4e52afb82496e4554101', 'request_id': 'req-ea040ea3-a011-41b1-86d5-f0b40295e1b6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.089003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46f05e0ab37c4e52afb82496e4554101', 'request_id': 'req-ea040ea3-a011-41b1-86d5-f0b40295e1b6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.102912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] The token cache is being invalidate because project 46f05e0ab37c4e52afb82496e4554101 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:10.103199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.103842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.105902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1074/2121] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:09 2022] DELETE /identity/v3/projects/46f05e0ab37c4e52afb82496e4554101 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:10.110401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:10.111973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:10.125418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e35ebef-fc99-47b9-b83e-73d46e28cce8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:10.183279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:10.183914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:10.184189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:10.184455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] PATH_INFO: `/v3/users/447a8c4337924918b0572a405f6ad868` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:10.185899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=447a8c4337924918b0572a405f6ad868)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:10.187843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:10.238612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '447a8c4337924918b0572a405f6ad868'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.243975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:10.247277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '447a8c4337924918b0572a405f6ad868'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.256397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '447a8c4337924918b0572a405f6ad868'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.257366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '447a8c4337924918b0572a405f6ad868'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.258340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-61e9e80d-2ab3-4077-a761-740d1a741fe7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '447a8c4337924918b0572a405f6ad868'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.302394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1048/2122] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:10 2022] DELETE /identity/v3/users/447a8c4337924918b0572a405f6ad868 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:10.538842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:10.540349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:10.557243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea040ea3-a011-41b1-86d5-f0b40295e1b6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:10.594421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:10.646463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:10.647219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:10.647625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:10.647874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] PATH_INFO: `/v3/projects/e4177fdf3e6e41afabcc047fe5caf7ad` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:10.649352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4177fdf3e6e41afabcc047fe5caf7ad)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:10.651612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:10.713596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4177fdf3e6e41afabcc047fe5caf7ad', 'request_id': 'req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.720128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4177fdf3e6e41afabcc047fe5caf7ad', 'request_id': 'req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.726393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4177fdf3e6e41afabcc047fe5caf7ad', 'request_id': 'req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.742115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] The token cache is being invalidate because project e4177fdf3e6e41afabcc047fe5caf7ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:10.742455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.743102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8cfc4f29-8745-4da9-81f3-e1fd3ffa9f13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:10.744968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1075/2123] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:10 2022] DELETE /identity/v3/projects/e4177fdf3e6e41afabcc047fe5caf7ad => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:10.750559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47264430-3f96-49c5-b422-204b91cd29cc None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:10.751091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47264430-3f96-49c5-b422-204b91cd29cc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:10.751091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47264430-3f96-49c5-b422-204b91cd29cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:10.775685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-47264430-3f96-49c5-b422-204b91cd29cc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:10.777300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-47264430-3f96-49c5-b422-204b91cd29cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:10.851503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1049/2124] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:10.859655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc466dc-af20-42f8-986e-063a3e4ec995 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:10.859996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc466dc-af20-42f8-986e-063a3e4ec995 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:10.860195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc466dc-af20-42f8-986e-063a3e4ec995 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:10.882079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-bdc466dc-af20-42f8-986e-063a3e4ec995 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:10.883325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bdc466dc-af20-42f8-986e-063a3e4ec995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:10.891345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1076/2125] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:10.896424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47264430-3f96-49c5-b422-204b91cd29cc None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:10.908059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47264430-3f96-49c5-b422-204b91cd29cc None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:10.913646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:10.914257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:10.914526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:10.914787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:10.915450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:10.915746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:10.917375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:10.933103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1050/2126] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:24:10.934935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc466dc-af20-42f8-986e-063a3e4ec995 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:10.945382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc466dc-af20-42f8-986e-063a3e4ec995 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:10.950775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:10.951390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:10.951652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:10.951915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:10.952615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:10.954691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:10.988981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1077/2127] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:10 2022] POST /identity/v3/projects => generated 375 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:10.993104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.003434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb282c3d-c6d4-4205-8a57-3e1995714599 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.073990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.074685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.075014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.075289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.075984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.078054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.103934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1051/2128] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:10 2022] POST /identity/v3/users => generated 288 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:11.109630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.122605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f077ef6d-3b70-40fd-8fbc-db8f6ecdd439 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.130958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.131801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.132142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.132484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.133476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.135930 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.147888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1078/2129] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:11.152160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.163945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93af6515-abbd-4e67-afde-690e948c4d1e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.170179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ea1990-4f91-4ed9-b2aa-7aced79f2e94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.170885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ea1990-4f91-4ed9-b2aa-7aced79f2e94 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.171255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ea1990-4f91-4ed9-b2aa-7aced79f2e94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.171578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ea1990-4f91-4ed9-b2aa-7aced79f2e94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.172242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ea1990-4f91-4ed9-b2aa-7aced79f2e94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.173881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ea1990-4f91-4ed9-b2aa-7aced79f2e94 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.186316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1052/2130] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:11.192961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.205357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8737fe84-760f-46e3-a65b-f3911177571b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.211466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.212115 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.212387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.212653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] PATH_INFO: `/v3/projects/518e67f987b24bc7b1834f5e51982732/users/5891cadab9cf4a779591450cfc957366/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.228596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=518e67f987b24bc7b1834f5e51982732, user_id=5891cadab9cf4a779591450cfc957366, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.231585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.246895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1079/2131] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:11 2022] PUT /identity/v3/projects/518e67f987b24bc7b1834f5e51982732/users/5891cadab9cf4a779591450cfc957366/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:11.260067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cad2877-468b-4679-a45d-4f9db338f9d8 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.260392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cad2877-468b-4679-a45d-4f9db338f9d8 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.260666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cad2877-468b-4679-a45d-4f9db338f9d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.290271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3cad2877-468b-4679-a45d-4f9db338f9d8 None None] MFA Rules not processed for user `5891cadab9cf4a779591450cfc957366`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:11.291506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3cad2877-468b-4679-a45d-4f9db338f9d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:11.318317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:11.373024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1053/2132] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:11.378217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.446487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:11.450897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4d6dee-12bd-4081-b600-8de492b2f5a1 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.456334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.457446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.457859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.458454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.459206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.461292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.494899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1080/2133] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:11 2022] POST /identity/v3/projects => generated 375 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:11.496043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cad2877-468b-4679-a45d-4f9db338f9d8 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.508279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cad2877-468b-4679-a45d-4f9db338f9d8 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.570142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.570786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.571103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.571389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.572097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.574360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.595982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1054/2134] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:11 2022] POST /identity/v3/users => generated 294 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:11.600071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.610602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a29feea-23c5-4250-960a-ba08f14ec9d4 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.616030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.616634 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.616932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.617233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.617899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.619522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.632197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1081/2135] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:11.636648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.647889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355277a5-b6da-403f-b1e1-be16a4a66700 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.654244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.654956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.655236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.655529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] PATH_INFO: `/v3/projects/ab3733e3f981479484b3f0444ced4e9b/users/b1296387b0374ca5b5e3189c0292673d/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.672361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab3733e3f981479484b3f0444ced4e9b, user_id=b1296387b0374ca5b5e3189c0292673d, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.675261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.692400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1055/2136] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:11 2022] PUT /identity/v3/projects/ab3733e3f981479484b3f0444ced4e9b/users/b1296387b0374ca5b5e3189c0292673d/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:11.696769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.707409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8287357-982b-4d94-9b64-21b303a1f457 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.777067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91eca66-fb5a-4c56-ae5e-71fc6274cf28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.777642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91eca66-fb5a-4c56-ae5e-71fc6274cf28 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.777924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91eca66-fb5a-4c56-ae5e-71fc6274cf28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.778187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91eca66-fb5a-4c56-ae5e-71fc6274cf28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.778881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91eca66-fb5a-4c56-ae5e-71fc6274cf28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.780456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91eca66-fb5a-4c56-ae5e-71fc6274cf28 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.790531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1082/2137] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:11.794898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.805958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1f17a9-0f12-482e-80b4-539f2e079633 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:11.811537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:11.812356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.812688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.813218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] PATH_INFO: `/v3/projects/ab3733e3f981479484b3f0444ced4e9b/users/b1296387b0374ca5b5e3189c0292673d/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.815386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab3733e3f981479484b3f0444ced4e9b, user_id=b1296387b0374ca5b5e3189c0292673d, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:11.818168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:11.830070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1056/2138] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:11 2022] PUT /identity/v3/projects/ab3733e3f981479484b3f0444ced4e9b/users/b1296387b0374ca5b5e3189c0292673d/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:11.841283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:11.841568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:11.841851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:11.871137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] MFA Rules not processed for user `b1296387b0374ca5b5e3189c0292673d`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:11.872478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:11.949064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1083/2139] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:11 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:11.962461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:11.964291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:12.026401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f15847c-652d-4702-9d3b-4706b64badc3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:12.030854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:12.031399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:12.031677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:12.031946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:12.065062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:12.095772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:12.097240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:12.107701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1057/2140] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:12.560278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:12.561876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:12.575778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71420877-b4c5-4829-a567-71c88ac60216 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:12.580416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937f6ec6-b0c4-4182-8554-07357d15a5ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:12.581148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f6ec6-b0c4-4182-8554-07357d15a5ed service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:12.581531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f6ec6-b0c4-4182-8554-07357d15a5ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:12.581887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f6ec6-b0c4-4182-8554-07357d15a5ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:12.584218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-937f6ec6-b0c4-4182-8554-07357d15a5ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:12.634298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:12.651414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937f6ec6-b0c4-4182-8554-07357d15a5ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:12.652850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937f6ec6-b0c4-4182-8554-07357d15a5ed service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:12.663027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1084/2141] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:12 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:21.035097 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:21.111661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c1d1c-9b58-40a6-aad2-b5683ff9cdeb service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:21.118640 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:21.119283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:21.119554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:21.119849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] PATH_INFO: `/v3/users/5891cadab9cf4a779591450cfc957366` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:21.120948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=5891cadab9cf4a779591450cfc957366)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:21.122917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:21.178715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5891cadab9cf4a779591450cfc957366'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.186044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5891cadab9cf4a779591450cfc957366'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.195365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5891cadab9cf4a779591450cfc957366'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.196036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5891cadab9cf4a779591450cfc957366'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.196802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5891cadab9cf4a779591450cfc957366'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.216738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1058/2142] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:21 2022] DELETE /identity/v3/users/5891cadab9cf4a779591450cfc957366 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:21.221414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:21.221932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:21.221932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:21.243338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:21.244723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:21.313664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1085/2143] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:21.324719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:21.326261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:21.395907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8472b0-cf6b-43bc-b7a2-3fe2e721cfaa admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:21.400014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4779bf4b-65f5-4877-b360-88b8444c0778 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:21.401304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779bf4b-65f5-4877-b360-88b8444c0778 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:21.401573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779bf4b-65f5-4877-b360-88b8444c0778 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:21.401855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779bf4b-65f5-4877-b360-88b8444c0778 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:21.409424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:21.415411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4779bf4b-65f5-4877-b360-88b8444c0778 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:21.417034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4779bf4b-65f5-4877-b360-88b8444c0778 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:21.434817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1059/2144] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:21.647392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:21.648958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:21.674100 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5080b-9431-482e-8a11-0f07f4db8221 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:21.683890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:21.684931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:21.685630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:21.686085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] PATH_INFO: `/v3/users/323456a5ac42467bbd4b1d368c87bde3` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:21.687831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=323456a5ac42467bbd4b1d368c87bde3)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:21.690721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:21.773703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '323456a5ac42467bbd4b1d368c87bde3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.800879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '323456a5ac42467bbd4b1d368c87bde3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.802539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '323456a5ac42467bbd4b1d368c87bde3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.818270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '323456a5ac42467bbd4b1d368c87bde3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.833178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '323456a5ac42467bbd4b1d368c87bde3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:21.839995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1086/2145] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:21 2022] DELETE /identity/v3/users/323456a5ac42467bbd4b1d368c87bde3 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:21.847890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:21.848227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:21.848494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:21.902955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:21.904042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:21.919515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:21.921301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:21.993161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1060/2146] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:22.004944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:22.007125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:22.012174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90d89f3-1ed7-403c-94c1-4aaa29f411b5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:22.017666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:22.018366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:22.018699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:22.019078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] PATH_INFO: `/v3/projects/518e67f987b24bc7b1834f5e51982732` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:22.020974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] RBAC: Authorizing `identity:delete_project(project_id=518e67f987b24bc7b1834f5e51982732)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:22.023521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:22.025305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:22.071540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb54968-5934-4aff-bf8e-21c93a310086 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:22.080418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '518e67f987b24bc7b1834f5e51982732', 'request_id': 'req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.086836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '518e67f987b24bc7b1834f5e51982732', 'request_id': 'req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.092756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '518e67f987b24bc7b1834f5e51982732', 'request_id': 'req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.106049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] The token cache is being invalidate because project 518e67f987b24bc7b1834f5e51982732 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:22.106329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.106959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.108690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1087/2147] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:21 2022] DELETE /identity/v3/projects/518e67f987b24bc7b1834f5e51982732 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:22.112394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:22.114203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:22.129838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:22.130466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:22.130865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:22.131155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:22.133053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:22.192253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb56c70-1cb2-4c9f-bd47-bebc2f37457b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:22.203893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:22.204544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:22.204819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:22.205087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] PATH_INFO: `/v3/users/b1296387b0374ca5b5e3189c0292673d` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:22.206451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1296387b0374ca5b5e3189c0292673d)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:22.209758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:22.210245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:22.213610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:22.220195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:22.226812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1061/2148] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:22.270422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1296387b0374ca5b5e3189c0292673d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.284928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1296387b0374ca5b5e3189c0292673d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.286086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1296387b0374ca5b5e3189c0292673d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.302517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1296387b0374ca5b5e3189c0292673d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.314758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1296387b0374ca5b5e3189c0292673d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.322323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1088/2149] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:22 2022] DELETE /identity/v3/users/b1296387b0374ca5b5e3189c0292673d => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:22.638812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:22.640249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:22.673809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f44046-a405-4f50-9abc-bc09032864a3 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:22.729897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:22.731389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:22.739788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:22.740438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:22.740698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:22.740971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] PATH_INFO: `/v3/projects/fcaeee6582514025bb9fa0e468128dd4` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:22.742214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fcaeee6582514025bb9fa0e468128dd4)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:22.744309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:22.745632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:22.800081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fcaeee6582514025bb9fa0e468128dd4', 'request_id': 'req-0d27b4af-14b1-464d-b432-ad8031f3f9a4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.815199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fcaeee6582514025bb9fa0e468128dd4', 'request_id': 'req-0d27b4af-14b1-464d-b432-ad8031f3f9a4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.816621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854206f-b2d7-4cc0-bce3-b14289d9ec9b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:22.822849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fcaeee6582514025bb9fa0e468128dd4', 'request_id': 'req-0d27b4af-14b1-464d-b432-ad8031f3f9a4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.834076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] The token cache is being invalidate because project fcaeee6582514025bb9fa0e468128dd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:22.834449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.835174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.837140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1062/2150] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:22 2022] DELETE /identity/v3/projects/fcaeee6582514025bb9fa0e468128dd4 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:22.840975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:22.842353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:22.875646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:22.876268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:22.876545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:22.876790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] PATH_INFO: `/v3/projects/ab3733e3f981479484b3f0444ced4e9b` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:22.877959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab3733e3f981479484b3f0444ced4e9b)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:22.880113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:22.907321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d27b4af-14b1-464d-b432-ad8031f3f9a4 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:22.912508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:22.913158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:22.913476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:22.913716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] PATH_INFO: `/v3/users/43f7ce5c18574055bb1e700a179e9310` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:22.914881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] RBAC: Authorizing `identity:delete_user(user_id=43f7ce5c18574055bb1e700a179e9310)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:22.917209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:22.939526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab3733e3f981479484b3f0444ced4e9b', 'request_id': 'req-88f39f5d-4e96-4607-b481-f404e5699e8e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.952447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab3733e3f981479484b3f0444ced4e9b', 'request_id': 'req-88f39f5d-4e96-4607-b481-f404e5699e8e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.960344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab3733e3f981479484b3f0444ced4e9b', 'request_id': 'req-88f39f5d-4e96-4607-b481-f404e5699e8e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.968573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:22.985798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] The token cache is being invalidate because project ab3733e3f981479484b3f0444ced4e9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:22.986178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.987047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-88f39f5d-4e96-4607-b481-f404e5699e8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.987967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43f7ce5c18574055bb1e700a179e9310'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:22.988951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1089/2151] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:22 2022] DELETE /identity/v3/projects/ab3733e3f981479484b3f0444ced4e9b => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:22.993683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f039365-afe2-492a-a28a-ca83fd19c98d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:22.994167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f039365-afe2-492a-a28a-ca83fd19c98d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:22.994273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f039365-afe2-492a-a28a-ca83fd19c98d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:22.997506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43f7ce5c18574055bb1e700a179e9310'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.008845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43f7ce5c18574055bb1e700a179e9310'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.009576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43f7ce5c18574055bb1e700a179e9310'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.010296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b1a816e5-8050-49d8-acab-5283bd115ffc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43f7ce5c18574055bb1e700a179e9310'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.016620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6f039365-afe2-492a-a28a-ca83fd19c98d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:23.017867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6f039365-afe2-492a-a28a-ca83fd19c98d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:23.041479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1063/2152] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:22 2022] DELETE /identity/v3/users/43f7ce5c18574055bb1e700a179e9310 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:23.080134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:23.103501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1090/2153] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:23.111003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e53a0-1ac1-4f00-a4ba-95407a45b34b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.111513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e53a0-1ac1-4f00-a4ba-95407a45b34b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.111594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e53a0-1ac1-4f00-a4ba-95407a45b34b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.133178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f36e53a0-1ac1-4f00-a4ba-95407a45b34b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:23.133676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f36e53a0-1ac1-4f00-a4ba-95407a45b34b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:23.235254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1064/2154] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:23.239762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f039365-afe2-492a-a28a-ca83fd19c98d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.249833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f039365-afe2-492a-a28a-ca83fd19c98d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.255257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.255902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.256180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.256451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.257139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:23.257453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.259071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.276293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1091/2155] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:24:23.281524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36e53a0-1ac1-4f00-a4ba-95407a45b34b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.293945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36e53a0-1ac1-4f00-a4ba-95407a45b34b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.301768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.302437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.302709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.302973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.303628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.305658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.339076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1065/2156] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/projects => generated 383 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:23.343694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.354265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fdb19d-0e37-4a18-92ff-d5058c538ef3 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.415907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.417985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:23.449867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.450602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.450942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.451275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.452165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.454273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.480061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1092/2157] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/users => generated 292 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:23.484922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.499857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc2712-8bd5-41e4-92f1-64f889cf433f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.504906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.505572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.505843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.506107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.506748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.508389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.513937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af28c107-3964-40ac-b9ac-cfca413d2c30 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.520116 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.521271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1093/2158] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:23 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:23.523866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.524289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.524596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] PATH_INFO: `/v3/projects/30d414797e99499984020d34de351fd6` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.525747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.526019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] RBAC: Authorizing `identity:delete_project(project_id=30d414797e99499984020d34de351fd6)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.527948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.539243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2c3354-83d0-4a58-94ff-9fa33506588b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.545562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.546177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.546450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.546716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.547364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.549091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.561624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1094/2159] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:23 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:24:23.565929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.575977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:23.577633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4be2b0-d50f-4b88-8272-8d4bbfb55b0c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.595950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30d414797e99499984020d34de351fd6', 'request_id': 'req-8b61cf6c-2371-4494-8c2c-1345e22fa47a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.606012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30d414797e99499984020d34de351fd6', 'request_id': 'req-8b61cf6c-2371-4494-8c2c-1345e22fa47a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.613804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30d414797e99499984020d34de351fd6', 'request_id': 'req-8b61cf6c-2371-4494-8c2c-1345e22fa47a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.626805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] The token cache is being invalidate because project 30d414797e99499984020d34de351fd6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:23.627913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.628683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8b61cf6c-2371-4494-8c2c-1345e22fa47a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:23.631010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1066/2160] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:23 2022] DELETE /identity/v3/projects/30d414797e99499984020d34de351fd6 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:23.636345 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dab2e3-a1fc-49fb-8567-d3316b17a767 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.636627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dab2e3-a1fc-49fb-8567-d3316b17a767 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.636975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dab2e3-a1fc-49fb-8567-d3316b17a767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.665278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f410d20-05b3-42a4-bb76-baf4223b90fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.666335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f410d20-05b3-42a4-bb76-baf4223b90fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.666658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f410d20-05b3-42a4-bb76-baf4223b90fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.667084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f410d20-05b3-42a4-bb76-baf4223b90fb admin admin] PATH_INFO: `/v3/projects/02331dc30f8445d4a6452c7033ff33d2/users/36fe8d4cbad742a197ea53836c2d1714/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.668090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-50dab2e3-a1fc-49fb-8567-d3316b17a767 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:23.669614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-50dab2e3-a1fc-49fb-8567-d3316b17a767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:23.682763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1067/2161] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:23.692158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3264f1-38c7-4066-8e26-45147b6abf35 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.692284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f410d20-05b3-42a4-bb76-baf4223b90fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=02331dc30f8445d4a6452c7033ff33d2, user_id=36fe8d4cbad742a197ea53836c2d1714, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.692449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3264f1-38c7-4066-8e26-45147b6abf35 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.692772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3264f1-38c7-4066-8e26-45147b6abf35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.695251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f410d20-05b3-42a4-bb76-baf4223b90fb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.721889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1095/2162] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:23 2022] PUT /identity/v3/projects/02331dc30f8445d4a6452c7033ff33d2/users/36fe8d4cbad742a197ea53836c2d1714/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:23.723422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4b3264f1-38c7-4066-8e26-45147b6abf35 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:23.724687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4b3264f1-38c7-4066-8e26-45147b6abf35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:23.734869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.735177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.735836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.773544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] MFA Rules not processed for user `36fe8d4cbad742a197ea53836c2d1714`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:23.774829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:23.792053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:23.820318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1068/2163] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:23.822283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3264f1-38c7-4066-8e26-45147b6abf35 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.835892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3264f1-38c7-4066-8e26-45147b6abf35 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.843983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.844641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.844911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.845199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.845869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:23.846169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.847847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.867665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1069/2164] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:24:23.872342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.876273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1096/2165] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:23.889857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.890577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f079a-1851-483a-96d8-3e18731764ec admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:23.891260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:23.896591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:23.897246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:23.897522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:23.897829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:23.898421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:23.900655 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:23.944964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1070/2166] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/projects => generated 381 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:23.949322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:23.966139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd009e69-9473-4d18-94de-b05702df0439 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.036476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.037101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.037474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.037776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.050278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76a9c69-944c-4dda-9a3f-910c75f65f46 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.055592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.056300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.056644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.056977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.057720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.059750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.096923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1071/2167] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:23 2022] POST /identity/v3/users => generated 291 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:24.099878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.111409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.112954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.113488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c670590-56dc-4527-9232-0f16bf8e608c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.118511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.119118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.119390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.120472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.120855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.122485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.123994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:24.127620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1097/2168] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:24.128796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.131229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:24.137986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1072/2169] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:24 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:24.139852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.147358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5857d7bc-c166-4724-93a8-ea9bdff8353d service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.152127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.152762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.153038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.153341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] PATH_INFO: `/v3/users/a8d008b266564a31a2f869f081389dc0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.154538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8d008b266564a31a2f869f081389dc0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.156447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.159645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d549c07-0e98-4b9d-b22f-0ccdd263ead0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.167143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.167883 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.168162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.168455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.169661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.173791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.198654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1073/2170] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:24 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:24.200427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.206350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:24.217809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35266ef2-51ef-4d2d-abe3-b808851249f7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.230580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb51941f-f76e-4144-86cc-430a6f0ac1dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.232330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51941f-f76e-4144-86cc-430a6f0ac1dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.233156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51941f-f76e-4144-86cc-430a6f0ac1dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.233439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb51941f-f76e-4144-86cc-430a6f0ac1dd admin admin] PATH_INFO: `/v3/projects/6e0b81deb0214359ad7e2425e134ab3a/users/815c24e9abe849f0833fe300aa1804a8/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.246597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8d008b266564a31a2f869f081389dc0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.262032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb51941f-f76e-4144-86cc-430a6f0ac1dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e0b81deb0214359ad7e2425e134ab3a, user_id=815c24e9abe849f0833fe300aa1804a8, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.265044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb51941f-f76e-4144-86cc-430a6f0ac1dd admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.276089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8d008b266564a31a2f869f081389dc0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.277028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8d008b266564a31a2f869f081389dc0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.283750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1074/2171] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:24 2022] PUT /identity/v3/projects/6e0b81deb0214359ad7e2425e134ab3a/users/815c24e9abe849f0833fe300aa1804a8/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:24.296274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8d008b266564a31a2f869f081389dc0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.296902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.297930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.298336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.306064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-8c66f5c5-e155-4c8c-b39e-340bbb6e3756 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8d008b266564a31a2f869f081389dc0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.312200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1098/2172] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:24 2022] DELETE /identity/v3/users/a8d008b266564a31a2f869f081389dc0 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:24.316646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.317460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.317909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.340519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] MFA Rules not processed for user `815c24e9abe849f0833fe300aa1804a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:24.342360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:24.346394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:24.348056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:24.438191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1075/2173] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:24 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:24.447623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1099/2174] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:24.458862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.459154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.460964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:24.461631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:24.577245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee78513-dc6d-439d-9df9-6b38780f8dbc None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.577525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e596c4c-102a-4151-b19c-80a7b6e08b74 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.581576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.582349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.582717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.582872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2827c94-1333-43a1-9a66-584bf36dec6b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.583026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.583461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2827c94-1333-43a1-9a66-584bf36dec6b service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.583830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2827c94-1333-43a1-9a66-584bf36dec6b service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.585672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2827c94-1333-43a1-9a66-584bf36dec6b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.595092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.597050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.599970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2827c94-1333-43a1-9a66-584bf36dec6b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.602894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2827c94-1333-43a1-9a66-584bf36dec6b service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.609462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1100/2175] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:24.614691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1076/2176] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:24 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:24.650481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.652420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:24.674432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd3838a-f7dc-4250-9017-4e47f2e26116 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:24.681578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:24.682701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.683211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.683686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] PATH_INFO: `/v3/users/815c24e9abe849f0833fe300aa1804a8` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.685282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=815c24e9abe849f0833fe300aa1804a8)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:24.687422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:24.738392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:24.753777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '815c24e9abe849f0833fe300aa1804a8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.770475 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '815c24e9abe849f0833fe300aa1804a8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.778162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '815c24e9abe849f0833fe300aa1804a8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.791595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '815c24e9abe849f0833fe300aa1804a8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.803098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '815c24e9abe849f0833fe300aa1804a8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:24.811167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1101/2177] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:24 2022] DELETE /identity/v3/users/815c24e9abe849f0833fe300aa1804a8 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:24.816504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:24.817684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:24.817684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:24.841014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:24.842395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:24.925316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:24.947545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1077/2178] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:24.960921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:24.962483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:25.019737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:25.022873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:25.038583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e4ee47-1a23-42d9-b04f-be77b4d26394 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:25.056711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:25.057405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:25.058184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:25.058464 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] PATH_INFO: `/v3/projects/363ab7ac0b0c4f37a92d6d4fa8359e8f` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:25.059734 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] RBAC: Authorizing `identity:delete_project(project_id=363ab7ac0b0c4f37a92d6d4fa8359e8f)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:25.062634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:25.078287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e207c6fc-276a-44ea-ba54-8d169a6f3a05 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:25.084255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:25.085007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:25.085319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:25.085611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:25.095964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:25.097411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:25.109867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1102/2179] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:25.128068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '363ab7ac0b0c4f37a92d6d4fa8359e8f', 'request_id': 'req-c698e9f3-b58e-46e2-81bf-d992f4840a5d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.136474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '363ab7ac0b0c4f37a92d6d4fa8359e8f', 'request_id': 'req-c698e9f3-b58e-46e2-81bf-d992f4840a5d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.142985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '363ab7ac0b0c4f37a92d6d4fa8359e8f', 'request_id': 'req-c698e9f3-b58e-46e2-81bf-d992f4840a5d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.153612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] The token cache is being invalidate because project 363ab7ac0b0c4f37a92d6d4fa8359e8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:25.153903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.155034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.156524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1078/2180] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:25 2022] DELETE /identity/v3/projects/363ab7ac0b0c4f37a92d6d4fa8359e8f => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:25.160637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:25.171245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeb1740-eeb8-46d3-86ae-29b98e15e987 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:25.234658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:25.235479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:25.235761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:25.236030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] PATH_INFO: `/v3/users/6111552258514603954e3e869f8eec0e` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:25.237235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6111552258514603954e3e869f8eec0e)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:25.239415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:25.299106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6111552258514603954e3e869f8eec0e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.306354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:25.312496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6111552258514603954e3e869f8eec0e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.313346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6111552258514603954e3e869f8eec0e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.323449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6111552258514603954e3e869f8eec0e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.332694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6111552258514603954e3e869f8eec0e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.340791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1103/2181] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:25 2022] DELETE /identity/v3/users/6111552258514603954e3e869f8eec0e => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:25.748127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:25.749731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:25.756683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:25.758240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:25.788604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c698e9f3-b58e-46e2-81bf-d992f4840a5d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:25.887048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284eeb02-a1a4-4440-bd50-5e2e95b08aa2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:25.897961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:25.898614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:25.898931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:25.899458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] PATH_INFO: `/v3/projects/6e0b81deb0214359ad7e2425e134ab3a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:25.900669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e0b81deb0214359ad7e2425e134ab3a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:25.903113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:25.909749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:25.911096 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:25.911395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:25.912106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] PATH_INFO: `/v3/projects/8c571653728b4cb48c108b13f3aa314b` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:25.913591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c571653728b4cb48c108b13f3aa314b)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:25.916446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:25.972769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e0b81deb0214359ad7e2425e134ab3a', 'request_id': 'req-17c9107e-3a08-4285-b79c-9cfadd1ca443'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.981083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e0b81deb0214359ad7e2425e134ab3a', 'request_id': 'req-17c9107e-3a08-4285-b79c-9cfadd1ca443'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:25.989643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e0b81deb0214359ad7e2425e134ab3a', 'request_id': 'req-17c9107e-3a08-4285-b79c-9cfadd1ca443'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.008394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] The token cache is being invalidate because project 6e0b81deb0214359ad7e2425e134ab3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:26.009052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.009504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c571653728b4cb48c108b13f3aa314b', 'request_id': 'req-f493ae1e-7847-4a21-bcf7-b63016951fd0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.010008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-17c9107e-3a08-4285-b79c-9cfadd1ca443 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.015433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1104/2182] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:25 2022] DELETE /identity/v3/projects/6e0b81deb0214359ad7e2425e134ab3a => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:26.019067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652e7d3-cb6e-4601-934e-e72ed0c053fa None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.019584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652e7d3-cb6e-4601-934e-e72ed0c053fa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.019909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a652e7d3-cb6e-4601-934e-e72ed0c053fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.025205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c571653728b4cb48c108b13f3aa314b', 'request_id': 'req-f493ae1e-7847-4a21-bcf7-b63016951fd0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.033152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c571653728b4cb48c108b13f3aa314b', 'request_id': 'req-f493ae1e-7847-4a21-bcf7-b63016951fd0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.045771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] The token cache is being invalidate because project 8c571653728b4cb48c108b13f3aa314b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:26.046090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.046897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f493ae1e-7847-4a21-bcf7-b63016951fd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:26.053179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1079/2183] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:25 2022] DELETE /identity/v3/projects/8c571653728b4cb48c108b13f3aa314b => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:26.055080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caca1585-5568-4729-973d-a9a8eb4ef106 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.055450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caca1585-5568-4729-973d-a9a8eb4ef106 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.055963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caca1585-5568-4729-973d-a9a8eb4ef106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.059885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a652e7d3-cb6e-4601-934e-e72ed0c053fa None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:26.061442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a652e7d3-cb6e-4601-934e-e72ed0c053fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:26.103658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:26.114624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-caca1585-5568-4729-973d-a9a8eb4ef106 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:26.135966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-caca1585-5568-4729-973d-a9a8eb4ef106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:26.170449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1105/2184] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:26.172119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a5ad47-d037-45a0-a4a5-9f661a8458c2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.172516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a5ad47-d037-45a0-a4a5-9f661a8458c2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.172798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a5ad47-d037-45a0-a4a5-9f661a8458c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.186176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:26.208486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-21a5ad47-d037-45a0-a4a5-9f661a8458c2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:26.212524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-21a5ad47-d037-45a0-a4a5-9f661a8458c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:26.229184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1106/2185] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:26.231340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a5ad47-d037-45a0-a4a5-9f661a8458c2 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.237880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1080/2186] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:26.245745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a5ad47-d037-45a0-a4a5-9f661a8458c2 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.254395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.254395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.254395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.254395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.258273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:26.258273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.258273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.258485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f27bd-766a-4c3a-9c29-e49f28103da9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.258485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f27bd-766a-4c3a-9c29-e49f28103da9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.258485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2f27bd-766a-4c3a-9c29-e49f28103da9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.271104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1107/2187] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:24:26.275687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.285268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1d2f27bd-766a-4c3a-9c29-e49f28103da9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:26.286414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d24cb3-72c3-4218-8726-724d77e017a8 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.286984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1d2f27bd-766a-4c3a-9c29-e49f28103da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:26.291925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.292900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.293286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.294109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.294372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.296630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.303915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1081/2188] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:26.308123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2f27bd-766a-4c3a-9c29-e49f28103da9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.319368 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2f27bd-766a-4c3a-9c29-e49f28103da9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.326653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.327320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.327598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.327867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.328535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:26.328859 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.329375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1108/2189] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:26.333287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.334507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.352569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1082/2190] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:24:26.353614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.365707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aebbdc1-74eb-402b-bd13-443907e4907d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.407147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfa6644-d4aa-45f5-8ec6-654b72cebad0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.412599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.413507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.413921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.414129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.414937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.417247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.423216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.423927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.424307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.424657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.425407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.427362 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.445555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1109/2191] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/users => generated 293 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:26.447110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.460260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a917b809-6931-4220-bcff-ded5f46c419b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.465523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1083/2192] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/projects => generated 377 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:26.466665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.539449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d2bd71-2326-43f2-9f4d-686e209586b3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.551957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.552582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.552853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.553144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.553314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.553789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.553973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.554263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.555020 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.555573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.555688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.558463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.576795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1110/2193] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:24:26.580945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.592470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb38d00-cf7b-49ff-bf2b-3dcff946baf0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.596626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1084/2194] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/users => generated 289 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:26.598682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.600290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.600981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.601349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.601820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.602533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.603971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:26.604260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.612302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fdd5bb-eb00-4d3e-a216-979d7645d9ea admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.620439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.621483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.621762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.622036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.623245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1111/2195] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:24:26.623612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.626005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.630245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.641803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1085/2196] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:26.642064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8312d29-df78-46e9-ab2c-85953fd544f6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.646994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.650170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f946be0c-6ad3-4c3a-a901-99cc12a246c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.650170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f946be0c-6ad3-4c3a-a901-99cc12a246c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.650170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f946be0c-6ad3-4c3a-a901-99cc12a246c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.650170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f946be0c-6ad3-4c3a-a901-99cc12a246c1 admin admin] PATH_INFO: `/v3/projects/b04c0b5bdb9248f0b7beda7814bd7638/users/12c3551297804416906459592b33b70a/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.669210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8b91d-6324-4c62-8f23-2e6a62e89de1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.673751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f946be0c-6ad3-4c3a-a901-99cc12a246c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b04c0b5bdb9248f0b7beda7814bd7638, user_id=12c3551297804416906459592b33b70a, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.675939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.676616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f946be0c-6ad3-4c3a-a901-99cc12a246c1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.676948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.677318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.677653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.678745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.681410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.698708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1086/2197] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:26.702501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1112/2198] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:26 2022] PUT /identity/v3/projects/b04c0b5bdb9248f0b7beda7814bd7638/users/12c3551297804416906459592b33b70a/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:26.710794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.713856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b091d9-f696-4473-ba3e-a16cdc1c9cb8 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.714194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b091d9-f696-4473-ba3e-a16cdc1c9cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.714544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b091d9-f696-4473-ba3e-a16cdc1c9cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.733801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8debff85-8f96-459e-add6-f734d8d7f86c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.753644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:26.758301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b5b091d9-f696-4473-ba3e-a16cdc1c9cb8 None None] MFA Rules not processed for user `12c3551297804416906459592b33b70a`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:26.760409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b5b091d9-f696-4473-ba3e-a16cdc1c9cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:26.807159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40b4988-855b-4517-a53c-03850105d481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.807832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b4988-855b-4517-a53c-03850105d481 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.808106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b4988-855b-4517-a53c-03850105d481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.808374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b4988-855b-4517-a53c-03850105d481 admin admin] PATH_INFO: `/v3/projects/a895b94351214ca6a67e82cb34915530/users/05a8c13e33e44c6585424325f7420fee/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.829589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40b4988-855b-4517-a53c-03850105d481 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a895b94351214ca6a67e82cb34915530, user_id=05a8c13e33e44c6585424325f7420fee, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.832445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40b4988-855b-4517-a53c-03850105d481 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.845933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1113/2199] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:26.851831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b091d9-f696-4473-ba3e-a16cdc1c9cb8 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:26.866298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1087/2200] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:26 2022] PUT /identity/v3/projects/a895b94351214ca6a67e82cb34915530/users/05a8c13e33e44c6585424325f7420fee/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:26.866502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b091d9-f696-4473-ba3e-a16cdc1c9cb8 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:26.880335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.881076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.881771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.925907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] MFA Rules not processed for user `05a8c13e33e44c6585424325f7420fee`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:26.928065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:26.942613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:26.943441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:26.943742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:26.944035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:26.944716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:26.947036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:26.991600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1114/2201] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/projects => generated 385 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:26.995342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.031725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1088/2202] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:27.044181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.045997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:27.110444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274991c3-af28-410c-aec0-552165aa3e79 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.116604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.117365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.117646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.118618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.119339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.122681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.139638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3a2711-883a-4d35-8782-ef58b6f338f3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.146206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:27.181361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1115/2203] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:26 2022] POST /identity/v3/users => generated 293 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:27.189306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.206239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c962a20-9786-4878-9821-caeef7d4453e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.220092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.220746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.221336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.221907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.222201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.223914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.239284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998503d-4524-44d1-a56d-ff10f84fad7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.242303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998503d-4524-44d1-a56d-ff10f84fad7b service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.242605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1116/2204] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:27.242954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998503d-4524-44d1-a56d-ff10f84fad7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.243240 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998503d-4524-44d1-a56d-ff10f84fad7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.251221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.265294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f600cfc-7cec-4a7b-a5da-c1ef5e12a1ca admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.272670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.273429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.273777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.274147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.274945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.276616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.278676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:27.295605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1117/2205] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:27.296559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.308571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93286636-fee5-40e3-a0ea-26c83218a720 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.314160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.315050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.315615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.315960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] PATH_INFO: `/v3/projects/67b547a04efd48c8be4cd109d26f31c7/users/1e776e5abd3c4a39871712acae8a2e89/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.327451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998503d-4524-44d1-a56d-ff10f84fad7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.328934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998503d-4524-44d1-a56d-ff10f84fad7b service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.346032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67b547a04efd48c8be4cd109d26f31c7, user_id=1e776e5abd3c4a39871712acae8a2e89, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.351320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.353565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1089/2206] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:27 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:27.384888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1118/2207] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:27 2022] PUT /identity/v3/projects/67b547a04efd48c8be4cd109d26f31c7/users/1e776e5abd3c4a39871712acae8a2e89/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:27.404584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb7e43a-c90e-4040-ac04-92755718dc7c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.404863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb7e43a-c90e-4040-ac04-92755718dc7c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.405535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb7e43a-c90e-4040-ac04-92755718dc7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.444697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-bdb7e43a-c90e-4040-ac04-92755718dc7c None None] MFA Rules not processed for user `1e776e5abd3c4a39871712acae8a2e89`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:27.445945 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bdb7e43a-c90e-4040-ac04-92755718dc7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:27.578427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1090/2208] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:27 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:27.583494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.687083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8c344d-53a6-4913-921d-3508d07d3027 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.694416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb96574-a054-418c-80e8-86d47a0e272b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.695049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb96574-a054-418c-80e8-86d47a0e272b admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.695328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb96574-a054-418c-80e8-86d47a0e272b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.695593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb96574-a054-418c-80e8-86d47a0e272b admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.696280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb96574-a054-418c-80e8-86d47a0e272b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.698404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb96574-a054-418c-80e8-86d47a0e272b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.734008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1119/2209] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:27 2022] POST /identity/v3/projects => generated 385 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:27.739102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb7e43a-c90e-4040-ac04-92755718dc7c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.757333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb7e43a-c90e-4040-ac04-92755718dc7c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.852969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:27.868344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265185a7-dd70-4c98-acbf-e3209392b1d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.870216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265185a7-dd70-4c98-acbf-e3209392b1d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.870216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265185a7-dd70-4c98-acbf-e3209392b1d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.870549 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265185a7-dd70-4c98-acbf-e3209392b1d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.871594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265185a7-dd70-4c98-acbf-e3209392b1d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.875388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265185a7-dd70-4c98-acbf-e3209392b1d6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.904033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e67fd-a212-4c59-ad0b-9f353b2d36cb None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.904397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e67fd-a212-4c59-ad0b-9f353b2d36cb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.904720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e67fd-a212-4c59-ad0b-9f353b2d36cb None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.906059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1120/2210] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:24:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 27 02:24:27.920007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1091/2211] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:27 2022] POST /identity/v3/users => generated 299 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:27.920910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67e67fd-a212-4c59-ad0b-9f353b2d36cb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.934636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67e67fd-a212-4c59-ad0b-9f353b2d36cb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.939269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.939877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.940176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.940724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.941373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:27.943126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:27.946192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d764250b-17bb-433f-87fa-5bb4ee6172bd None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.946465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d764250b-17bb-433f-87fa-5bb4ee6172bd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.946730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d764250b-17bb-433f-87fa-5bb4ee6172bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:27.960311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1121/2212] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:24:27.964181 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:27.973598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-d764250b-17bb-433f-87fa-5bb4ee6172bd None None] MFA Rules not processed for user `fa1922df0af7489db403cb6e42a7311b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:27.976148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d764250b-17bb-433f-87fa-5bb4ee6172bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:27.976314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42e5499-75c2-4e9e-9bca-9840ef25c91e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:27.984403 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:27.984688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:27.985061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:27.985357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] PATH_INFO: `/v3/projects/34d36583230e4b7585657541e355efdd/users/15c3b558609a4432bde2ead23d1ca690/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.015600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34d36583230e4b7585657541e355efdd, user_id=15c3b558609a4432bde2ead23d1ca690, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:28.021879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:28.042896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1122/2213] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:27 2022] PUT /identity/v3/projects/34d36583230e4b7585657541e355efdd/users/15c3b558609a4432bde2ead23d1ca690/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:28.043753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:28.045498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.078326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1092/2214] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:24:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:28.079850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d764250b-17bb-433f-87fa-5bb4ee6172bd None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:28.091648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d764250b-17bb-433f-87fa-5bb4ee6172bd None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:28.128230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1cb5e-68af-4096-bb4e-b0444abbbea0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:28.136950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:28.137640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:28.139781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:28.140815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] PATH_INFO: `/v3/users/55640f750d1b49be9c3f79501a63c1ab` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.143307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55640f750d1b49be9c3f79501a63c1ab)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:28.146552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:28.168984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:28.170101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:28.170101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:28.170305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.170885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:28.172699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:28.174631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:28.198753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1093/2215] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:28 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:28.201331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:28.203601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.214712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55640f750d1b49be9c3f79501a63c1ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:28.232681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55640f750d1b49be9c3f79501a63c1ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:28.233863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55640f750d1b49be9c3f79501a63c1ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:28.243933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55640f750d1b49be9c3f79501a63c1ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:28.254565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55640f750d1b49be9c3f79501a63c1ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:28.262251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1123/2216] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:28 2022] DELETE /identity/v3/users/55640f750d1b49be9c3f79501a63c1ab => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:28.263535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:28.266715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.316690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f77c5a-8372-4b1d-b24d-ef6ece292bc2 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:28.331614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:28.345102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546d424c-5450-42d2-87cb-4bb3a484d371 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:28.360466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bc01ec-dd07-419c-a43d-02efdaebaaf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:28.361262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bc01ec-dd07-419c-a43d-02efdaebaaf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:28.361544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bc01ec-dd07-419c-a43d-02efdaebaaf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:28.361879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bc01ec-dd07-419c-a43d-02efdaebaaf8 admin admin] PATH_INFO: `/v3/projects/34d36583230e4b7585657541e355efdd/users/15c3b558609a4432bde2ead23d1ca690/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.364225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bc01ec-dd07-419c-a43d-02efdaebaaf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34d36583230e4b7585657541e355efdd, user_id=15c3b558609a4432bde2ead23d1ca690, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:28.366925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bc01ec-dd07-419c-a43d-02efdaebaaf8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:28.378935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1124/2217] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:28 2022] PUT /identity/v3/projects/34d36583230e4b7585657541e355efdd/users/15c3b558609a4432bde2ead23d1ca690/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:28.389204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:28.389526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:28.389895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.406860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:28.407621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:28.407966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:28.408218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.410603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.419948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] MFA Rules not processed for user `15c3b558609a4432bde2ead23d1ca690`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:28.421754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.424993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:28.426792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:28.438079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1094/2218] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:28.520089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1125/2219] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:28 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:28.533774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:28.535805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.598115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b656a34-55f3-4841-b328-3089a067b587 service placement] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:28.653781 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efe38b5-dd0b-40d0-9333-e5052459b76d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:28.654401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efe38b5-dd0b-40d0-9333-e5052459b76d service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:28.654656 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efe38b5-dd0b-40d0-9333-e5052459b76d service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:28.654937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efe38b5-dd0b-40d0-9333-e5052459b76d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.657248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4efe38b5-dd0b-40d0-9333-e5052459b76d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.730276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efe38b5-dd0b-40d0-9333-e5052459b76d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:28.731853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efe38b5-dd0b-40d0-9333-e5052459b76d service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:28.742494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1095/2220] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:28.763602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:28.765715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:28.821054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:28.892296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c55ac3-c0fd-437b-84bf-6cba4984ec13 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:28.899269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:28.899925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:28.900198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:28.900462 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] PATH_INFO: `/v3/projects/f7ef4b44a3f94d599eacce269b06594c` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:28.901776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7ef4b44a3f94d599eacce269b06594c)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:28.903830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:28.969631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7ef4b44a3f94d599eacce269b06594c', 'request_id': 'req-3006d995-7d86-4adb-a208-b04b7a96f761'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:28.978454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7ef4b44a3f94d599eacce269b06594c', 'request_id': 'req-3006d995-7d86-4adb-a208-b04b7a96f761'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:28.986310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7ef4b44a3f94d599eacce269b06594c', 'request_id': 'req-3006d995-7d86-4adb-a208-b04b7a96f761'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:29.003795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] The token cache is being invalidate because project f7ef4b44a3f94d599eacce269b06594c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:29.004129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:29.004854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-3006d995-7d86-4adb-a208-b04b7a96f761 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:29.006667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1126/2221] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:28 2022] DELETE /identity/v3/projects/f7ef4b44a3f94d599eacce269b06594c => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:29.011617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbfd774-4b3a-4c84-9a6b-bb4892c8ecb8 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.011747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbfd774-4b3a-4c84-9a6b-bb4892c8ecb8 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.012021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbfd774-4b3a-4c84-9a6b-bb4892c8ecb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.037305 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9cbfd774-4b3a-4c84-9a6b-bb4892c8ecb8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:29.038843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9cbfd774-4b3a-4c84-9a6b-bb4892c8ecb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:29.052725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:29.108329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1096/2222] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:29.116651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c586e3-efb3-47bb-822d-c0916924a735 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.117332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c586e3-efb3-47bb-822d-c0916924a735 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.117608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c586e3-efb3-47bb-822d-c0916924a735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.140920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-42c586e3-efb3-47bb-822d-c0916924a735 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:29.142423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-42c586e3-efb3-47bb-822d-c0916924a735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:29.151865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1127/2223] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:29.157421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbfd774-4b3a-4c84-9a6b-bb4892c8ecb8 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:29.166720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbfd774-4b3a-4c84-9a6b-bb4892c8ecb8 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:29.171757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:29.172421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.173439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.173724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.174461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:29.174727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:29.177524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:29.193685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1097/2224] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:24:29.196859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c586e3-efb3-47bb-822d-c0916924a735 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:29.207492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c586e3-efb3-47bb-822d-c0916924a735 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:29.212730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:29.213425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.213703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.213972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.214618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:29.217101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:29.251630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1128/2225] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:29 2022] POST /identity/v3/projects => generated 387 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:29.258232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:29.270671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619ff0c1-47cb-4291-87ca-cce0615ddbb0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:29.336384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:29.337012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.337334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.337723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.338466 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:29.340350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:29.365793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1098/2226] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:29 2022] POST /identity/v3/users => generated 294 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:29.367764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:29.379193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b618ef41-fc9d-40d9-9d06-de8d13df5157 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:29.384370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:29.385101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.385413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.385682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.386338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:29.387971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:29.398723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1129/2227] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:29.402928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:29.413461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abfc33-aef0-4a30-a78f-525674a27626 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:29.420540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583345e3-f12f-4b82-92d6-263bcce7548e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:29.421175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583345e3-f12f-4b82-92d6-263bcce7548e admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.421453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583345e3-f12f-4b82-92d6-263bcce7548e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.421725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583345e3-f12f-4b82-92d6-263bcce7548e admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.422384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583345e3-f12f-4b82-92d6-263bcce7548e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:29.423985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583345e3-f12f-4b82-92d6-263bcce7548e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:29.435109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1099/2228] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:29.439390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:29.449991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85d4963-862f-4e91-99c8-925247813929 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:29.454558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:29.455213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.455489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.455760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] PATH_INFO: `/v3/projects/c938ae5fd22a43e8a01859da2a336b53/users/d750e6e172d34d4a9333e0c059c5442b/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.470947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c938ae5fd22a43e8a01859da2a336b53, user_id=d750e6e172d34d4a9333e0c059c5442b, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:29.474429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:29.490843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1130/2229] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:29 2022] PUT /identity/v3/projects/c938ae5fd22a43e8a01859da2a336b53/users/d750e6e172d34d4a9333e0c059c5442b/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:29.497986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.497986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.498454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.520995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:29.522241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:29.593999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1100/2230] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:29.604592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:29.606790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:29.660981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:29.667861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22dbb3-0ebe-4d5e-b9fa-1cacc798ec5f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:29.672055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:29.672719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:29.673136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:29.673425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:29.682882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:29.684311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:29.696356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1131/2231] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:31.549924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:31.553017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:31.589844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:31.626509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f48331-8d87-4f5c-9a56-76356db52a48 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:31.630571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2765cd5-94c8-4311-862c-edbc22e029ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:31.631252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2765cd5-94c8-4311-862c-edbc22e029ba service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:31.631525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2765cd5-94c8-4311-862c-edbc22e029ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:31.631800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2765cd5-94c8-4311-862c-edbc22e029ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:31.633574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e2765cd5-94c8-4311-862c-edbc22e029ba service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:31.704125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2765cd5-94c8-4311-862c-edbc22e029ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:31.705601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2765cd5-94c8-4311-862c-edbc22e029ba service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:31.717856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1101/2232] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:32.731243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:32.742142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d735da2-c328-4ab4-8acf-0c188068669d service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:32.745816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:32.746533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:32.746878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:32.747242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:32.749278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:32.829760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:32.831978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:32.846592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1132/2233] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:32 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:34.321401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18a872-8065-4717-b130-965cc9277ed8 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:34.322251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18a872-8065-4717-b130-965cc9277ed8 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:34.322737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18a872-8065-4717-b130-965cc9277ed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:34.358418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5c18a872-8065-4717-b130-965cc9277ed8 None None] MFA Rules not processed for user `d750e6e172d34d4a9333e0c059c5442b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:34.359519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5c18a872-8065-4717-b130-965cc9277ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:34.453928 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1102/2234] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:34 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:34.469199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:34.470726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:34.559819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3279b338-e98c-4a13-8d00-68da1a8c167d service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:34.566973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:34.567620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:34.567898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:34.568162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:34.577856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:34.579805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:34.589981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1133/2235] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:34 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:36.276583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d533b-5a76-49d7-b0ac-5a28f930d8b2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:36.277469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d533b-5a76-49d7-b0ac-5a28f930d8b2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:36.277469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d533b-5a76-49d7-b0ac-5a28f930d8b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:36.304227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e56d533b-5a76-49d7-b0ac-5a28f930d8b2 None None] MFA Rules not processed for user `d750e6e172d34d4a9333e0c059c5442b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:36.305925 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e56d533b-5a76-49d7-b0ac-5a28f930d8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:36.317844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1103/2236] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:36 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:36.542439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:36.544434 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:36.562461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615fe443-4105-4c93-bc39-244628eb702c service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:36.571938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:36.573164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:36.573595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:36.574414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] PATH_INFO: `/v3/users/36fe8d4cbad742a197ea53836c2d1714` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:36.575918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] RBAC: Authorizing `identity:delete_user(user_id=36fe8d4cbad742a197ea53836c2d1714)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:36.578855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:36.650786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36fe8d4cbad742a197ea53836c2d1714'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:36.694306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:36.697686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36fe8d4cbad742a197ea53836c2d1714'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:36.699377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36fe8d4cbad742a197ea53836c2d1714'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:36.727102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36fe8d4cbad742a197ea53836c2d1714'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:36.749738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36fe8d4cbad742a197ea53836c2d1714'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:36.767390 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1134/2237] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:36 2022] DELETE /identity/v3/users/36fe8d4cbad742a197ea53836c2d1714 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:36.773354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:36.773654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:36.774112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:36.795868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:36.797045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:36.886566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1104/2238] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:36.906226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:36.906226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:36.982129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b2552e-e644-4fbe-b4b9-98b2c38f6858 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:36.986095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e2bff5-d91f-4dcd-aaf3-9a55d9382d6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:36.986813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e2bff5-d91f-4dcd-aaf3-9a55d9382d6f service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:36.987182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e2bff5-d91f-4dcd-aaf3-9a55d9382d6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:36.987541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e2bff5-d91f-4dcd-aaf3-9a55d9382d6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:36.999951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e2bff5-d91f-4dcd-aaf3-9a55d9382d6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:37.001580 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e2bff5-d91f-4dcd-aaf3-9a55d9382d6f service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:37.014777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1135/2239] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:37.457164 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:37.459059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:37.466471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:37.478789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce09827-b67c-4ab3-b667-37f96a2c48d9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:37.487371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:37.489648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:37.490363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:37.490732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] PATH_INFO: `/v3/projects/02331dc30f8445d4a6452c7033ff33d2` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:37.493091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02331dc30f8445d4a6452c7033ff33d2)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:37.495890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:37.609188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02331dc30f8445d4a6452c7033ff33d2', 'request_id': 'req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:37.622157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02331dc30f8445d4a6452c7033ff33d2', 'request_id': 'req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:37.636696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02331dc30f8445d4a6452c7033ff33d2', 'request_id': 'req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:37.667255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] The token cache is being invalidate because project 02331dc30f8445d4a6452c7033ff33d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:37.667858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:37.668910 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d040c51-04e2-4ab5-acf6-a91ae96b1d24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:37.671022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1105/2240] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:37 2022] DELETE /identity/v3/projects/02331dc30f8445d4a6452c7033ff33d2 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:37.675940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba8c1b-e642-48f1-8322-3f1a1a1ce341 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:37.676675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba8c1b-e642-48f1-8322-3f1a1a1ce341 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:37.676675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba8c1b-e642-48f1-8322-3f1a1a1ce341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:37.734288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2fba8c1b-e642-48f1-8322-3f1a1a1ce341 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:37.735678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2fba8c1b-e642-48f1-8322-3f1a1a1ce341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:37.873790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1136/2241] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:37.880757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c514a79-4007-4c11-b289-131310fff793 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:37.881455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c514a79-4007-4c11-b289-131310fff793 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:37.881958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c514a79-4007-4c11-b289-131310fff793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:37.906093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5c514a79-4007-4c11-b289-131310fff793 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:37.907619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5c514a79-4007-4c11-b289-131310fff793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:37.917593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1106/2242] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:37.921790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba8c1b-e642-48f1-8322-3f1a1a1ce341 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:37.933786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba8c1b-e642-48f1-8322-3f1a1a1ce341 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:37.939308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:37.939960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:37.940190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:37.940652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:37.941467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:37.941819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:37.944135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:37.965956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c514a79-4007-4c11-b289-131310fff793 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:37.967784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1137/2243] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:24:37.980301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c514a79-4007-4c11-b289-131310fff793 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:37.985840 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:37.986388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:37.986997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:37.987163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:37.987843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:37.990324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:38.030710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1107/2244] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:37 2022] POST /identity/v3/projects => generated 371 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:38.031647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:38.042633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9f6aee-4349-457c-ab6a-6760d9c5ae88 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:38.109231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:38.109843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:38.110117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:38.110384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:38.111143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:38.115254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:38.136792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:38.143526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1138/2245] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:38 2022] POST /identity/v3/users => generated 286 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:38.147660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:38.160737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6a962d-d31a-4be3-8d96-7d31bfdc3d4c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:38.168227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:38.169787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:38.169938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:38.171730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:38.171730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:38.172629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:38.194146 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1108/2246] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:38 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:38.198085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:38.209535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb151af2-a468-40f0-83f4-abd5cf3f9191 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:38.214581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe005379-69ac-4524-84af-fde799f3951c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:38.215797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe005379-69ac-4524-84af-fde799f3951c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:38.215797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe005379-69ac-4524-84af-fde799f3951c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:38.215959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe005379-69ac-4524-84af-fde799f3951c admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:38.216590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe005379-69ac-4524-84af-fde799f3951c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:38.218754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe005379-69ac-4524-84af-fde799f3951c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:38.232663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1139/2247] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:38 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:38.236894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:38.248486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34450a72-3efa-4129-a872-8d401f5207c0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:38.253869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:38.255541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:38.255818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:38.256091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] PATH_INFO: `/v3/projects/b2ddff853b7546788b645a0344c2297b/users/642291c4daa04a6ab3fbce92a8be39c6/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:38.274905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2ddff853b7546788b645a0344c2297b, user_id=642291c4daa04a6ab3fbce92a8be39c6, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:38.277563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:38.297506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1109/2248] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:38 2022] PUT /identity/v3/projects/b2ddff853b7546788b645a0344c2297b/users/642291c4daa04a6ab3fbce92a8be39c6/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:38.300244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6cd313-244f-4346-835e-de65312b9cd7 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:38.300804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6cd313-244f-4346-835e-de65312b9cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:38.301195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6cd313-244f-4346-835e-de65312b9cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:38.323567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ec6cd313-244f-4346-835e-de65312b9cd7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:38.325664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ec6cd313-244f-4346-835e-de65312b9cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:38.398238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1140/2249] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:38.408145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:38.410188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:38.470558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d2e491-d8ef-426b-b60f-0bec5f31c841 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:38.476073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:38.476887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:38.477420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:38.477880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:38.491233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:38.492909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:38.504818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1110/2250] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:43.448797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e655111-43dc-41db-acc7-9640d1d845d0 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:43.459012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e655111-43dc-41db-acc7-9640d1d845d0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:43.463883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e655111-43dc-41db-acc7-9640d1d845d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:43.508714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4e655111-43dc-41db-acc7-9640d1d845d0 None None] MFA Rules not processed for user `642291c4daa04a6ab3fbce92a8be39c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:43.510117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4e655111-43dc-41db-acc7-9640d1d845d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:43.649260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1141/2251] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:43 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:43.654062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:43.670812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42279540-52f4-45ab-b641-f946f424040a service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:43.677651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:43.678552 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:43.678849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:43.679151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:43.679854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:43.682068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:43.732027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1111/2252] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:43 2022] POST /identity/v3/projects => generated 371 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:43.733215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e655111-43dc-41db-acc7-9640d1d845d0 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:43.744635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e655111-43dc-41db-acc7-9640d1d845d0 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:43.808485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:43.809136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:43.809423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:43.809692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:43.810377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:43.812346 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:43.839980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1142/2253] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:43 2022] POST /identity/v3/users => generated 292 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:24:43.844271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:43.851312 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:43.859034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0220623-b23e-4719-a336-9fa171e13749 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:43.868186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:43.868489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:43.868844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:43.869008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:43.869712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:43.871441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:43.885475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1112/2254] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:43 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:43.889976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:43.917559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc2bf88-5aa1-4145-a623-a276f3458325 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:43.928759 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:43.929880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:43.930188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:43.930464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] PATH_INFO: `/v3/projects/22afc26b2f4243289fd730ec223b2cca/users/399110b53734477c98318822e79eb390/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:43.936927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:43.952034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22afc26b2f4243289fd730ec223b2cca, user_id=399110b53734477c98318822e79eb390, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:43.954963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:43.968603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1143/2255] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:43 2022] PUT /identity/v3/projects/22afc26b2f4243289fd730ec223b2cca/users/399110b53734477c98318822e79eb390/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:43.974609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:43.987997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d12ed62-1619-4ece-acd4-245a4ba9555b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:44.052343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:44.053033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:44.053487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:44.053780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:44.058440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:44.060975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:44.077156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1113/2256] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:43 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:44.078428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:44.102092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4fcbdf-273a-430c-9af5-9ba5c07dc716 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:44.117402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:44.118509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:44.118854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:44.119636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] PATH_INFO: `/v3/projects/22afc26b2f4243289fd730ec223b2cca/users/399110b53734477c98318822e79eb390/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:44.121612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=22afc26b2f4243289fd730ec223b2cca, user_id=399110b53734477c98318822e79eb390, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:44.124749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:44.138618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1144/2257] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:44 2022] PUT /identity/v3/projects/22afc26b2f4243289fd730ec223b2cca/users/399110b53734477c98318822e79eb390/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:44.671994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:44.677894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:44.692664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e83241-d8ad-4418-84c1-2a44ee23d30f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:44.819879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:44.819879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:44.820117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:44.820297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] PATH_INFO: `/v3/users/420d058b274d4c0395163389b3a02aa8` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:44.823279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] RBAC: Authorizing `identity:delete_user(user_id=420d058b274d4c0395163389b3a02aa8)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:44.825630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:44.859294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:44.861305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:44.885947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891af17e-c99b-4a2f-8760-9f8ff5c62a8d admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:44.907048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:44.907961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:44.908385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:44.908803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] PATH_INFO: `/v3/users/12c3551297804416906459592b33b70a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:44.910513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12c3551297804416906459592b33b70a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:44.912046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '420d058b274d4c0395163389b3a02aa8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:44.914958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:44.922015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '420d058b274d4c0395163389b3a02aa8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:44.932707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '420d058b274d4c0395163389b3a02aa8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:44.934381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '420d058b274d4c0395163389b3a02aa8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:44.935782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-53d658c1-b87d-4c10-9a06-43a9e9d8a567 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '420d058b274d4c0395163389b3a02aa8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:44.963893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1114/2258] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:44 2022] DELETE /identity/v3/users/420d058b274d4c0395163389b3a02aa8 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:44.971202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:44.971825 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:44.972130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:44.997328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:45.014625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:45.019898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:45.038232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12c3551297804416906459592b33b70a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:45.068924 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12c3551297804416906459592b33b70a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:45.069851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12c3551297804416906459592b33b70a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:45.085717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12c3551297804416906459592b33b70a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:45.103340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-cf65a58e-e52a-4d87-a509-8ab1477180b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12c3551297804416906459592b33b70a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:45.110432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1145/2259] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:44 2022] DELETE /identity/v3/users/12c3551297804416906459592b33b70a => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:45.114825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:45.115193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:45.115538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:45.127542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1115/2260] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:45.142889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:45.144413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:45.148968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:45.150439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:45.235514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db31275c-435b-4b7a-bd40-44b66677d2d3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:45.286949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1146/2261] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:45.312571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:45.316587 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:45.340189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:45.340880 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:45.341254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:45.341571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:45.361465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:45.363843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:45.389586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1116/2262] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:45.449441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:45.492554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c16c8a9-fa7a-4fd5-950c-9777e6c69063 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:45.508489 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a1822e-879c-47fa-adfd-90f57dbb2c7b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:45.509772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1822e-879c-47fa-adfd-90f57dbb2c7b service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:45.512069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1822e-879c-47fa-adfd-90f57dbb2c7b service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:45.513902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1822e-879c-47fa-adfd-90f57dbb2c7b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:45.543983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a1822e-879c-47fa-adfd-90f57dbb2c7b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:45.547423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a1822e-879c-47fa-adfd-90f57dbb2c7b service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:45.580135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1147/2263] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:46.446187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:46.448471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:46.477860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad1cd8b-260a-41cb-9013-c087e0ab96bb service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:46.491144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:46.492793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:46.493204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:46.493766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] PATH_INFO: `/v3/projects/4f1fe8cae02a410b91854d368a4b959e` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:46.495292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f1fe8cae02a410b91854d368a4b959e)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:46.498632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:46.597800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f1fe8cae02a410b91854d368a4b959e', 'request_id': 'req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:46.619101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f1fe8cae02a410b91854d368a4b959e', 'request_id': 'req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:46.630528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f1fe8cae02a410b91854d368a4b959e', 'request_id': 'req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:46.632717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:46.653954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] The token cache is being invalidate because project 4f1fe8cae02a410b91854d368a4b959e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:46.654252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:46.655585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8f8d0312-5f2c-465b-b7b9-2a002c09e4e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:46.669590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1117/2264] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:46 2022] DELETE /identity/v3/projects/4f1fe8cae02a410b91854d368a4b959e => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:46.675296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:46.675789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:46.676028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:46.734609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:46.735902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:46.828770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1148/2265] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:46.836693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0286faee-20bc-496b-9baa-0abc88d0956d None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:46.837841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0286faee-20bc-496b-9baa-0abc88d0956d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:46.838878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0286faee-20bc-496b-9baa-0abc88d0956d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:46.892959 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-0286faee-20bc-496b-9baa-0abc88d0956d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:46.895565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0286faee-20bc-496b-9baa-0abc88d0956d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:46.903791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:46.905056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:46.919056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1118/2266] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:46.921262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0286faee-20bc-496b-9baa-0abc88d0956d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:46.922041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c34b0b-ef65-440f-9e7c-30643ed30152 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:46.938895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:46.939807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:46.940238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:46.940583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] PATH_INFO: `/v3/projects/b04c0b5bdb9248f0b7beda7814bd7638` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:46.944066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0286faee-20bc-496b-9baa-0abc88d0956d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:46.945132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b04c0b5bdb9248f0b7beda7814bd7638)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:46.949718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:46.974533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:46.975165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:46.975435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:46.975703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:46.976407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:46.977140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:46.999155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:47.043132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1119/2267] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:24:47.065174 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:47.103483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b76b13-6c9c-46ce-a364-2724d645b347 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:47.105589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b04c0b5bdb9248f0b7beda7814bd7638', 'request_id': 'req-5fdca748-0d84-4594-83d8-c7afa7e4fa15'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.128736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b04c0b5bdb9248f0b7beda7814bd7638', 'request_id': 'req-5fdca748-0d84-4594-83d8-c7afa7e4fa15'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.138231 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b04c0b5bdb9248f0b7beda7814bd7638', 'request_id': 'req-5fdca748-0d84-4594-83d8-c7afa7e4fa15'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.161792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] The token cache is being invalidate because project b04c0b5bdb9248f0b7beda7814bd7638 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:47.162271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.167640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.170811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1149/2268] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:46 2022] DELETE /identity/v3/projects/b04c0b5bdb9248f0b7beda7814bd7638 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:47.174246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:47.176372 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:47.235257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:47.235828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:47.236285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:47.236575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:47.237248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:47.239432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:47.281802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1120/2269] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:47 2022] POST /identity/v3/projects => generated 391 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:47.283846 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:47.285181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:47.344979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdca748-0d84-4594-83d8-c7afa7e4fa15 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:47.391771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85471f5e-e93e-4c72-abe1-bc6babb98e60 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:47.408336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:47.408336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:47.408624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:47.408858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:47.414188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:47.416740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:47.437783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:47.476271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1121/2270] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:47 2022] POST /identity/v3/users => generated 296 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:47.479074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:47.507257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6949ece6-9722-4cc3-abcd-e8235ef14e36 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:47.520268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:47.522471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:47.522956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:47.523691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:47.526709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:47.531773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:47.566478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1122/2271] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:47 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:47.568346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:47.595703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd20d8f-c879-4c6e-b11c-03b4dc4d2bec admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:47.625053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:47.625993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:47.626798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:47.629056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:47.629056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] PATH_INFO: `/v3/users/1e776e5abd3c4a39871712acae8a2e89` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:47.630110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:47.630385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:47.630966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e776e5abd3c4a39871712acae8a2e89)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:47.632068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:47.632526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:47.633941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:47.642439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:47.725087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1123/2272] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:47 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:47.733870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:47.757680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e776e5abd3c4a39871712acae8a2e89'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.780092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536993d-2a1e-4e17-a853-53d201495e35 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:47.781077 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e776e5abd3c4a39871712acae8a2e89'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.782093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e776e5abd3c4a39871712acae8a2e89'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.798385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e776e5abd3c4a39871712acae8a2e89'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.810006 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:47.810624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ce5ecd8a-54cc-4512-8725-ffb437e5019a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e776e5abd3c4a39871712acae8a2e89'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:47.810761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:47.810957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:47.811233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] PATH_INFO: `/v3/projects/faac909270b6414eabe3a4b144539f4d/users/ed17451121384ab1b8aa47a88ad9c5ae/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:47.830284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1150/2273] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:47 2022] DELETE /identity/v3/users/1e776e5abd3c4a39871712acae8a2e89 => generated 0 bytes in 656 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:47.868147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=faac909270b6414eabe3a4b144539f4d, user_id=ed17451121384ab1b8aa47a88ad9c5ae, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:47.873586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:47.884998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=2,keystone:SELECT=93,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:47.892929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1124/2274] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:47 2022] PUT /identity/v3/projects/faac909270b6414eabe3a4b144539f4d/users/ed17451121384ab1b8aa47a88ad9c5ae/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:47.904074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:47.904473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:47.904734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:47.940842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] MFA Rules not processed for user `ed17451121384ab1b8aa47a88ad9c5ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:47.942889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:48.041358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1151/2275] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:47 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:48.053654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:48.055606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:48.147651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa0cb4-93cf-4b29-b0e7-b16b709fad44 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:48.152508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:48.154034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:48.154938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:48.156046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:48.166242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:48.167778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:48.190730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1125/2276] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:48 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:48.803598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:48.807715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:48.919377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:48.993444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d20060-4224-46e3-98d9-202439c97801 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:49.015985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:49.017520 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:49.017898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:49.018635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] PATH_INFO: `/v3/projects/67b547a04efd48c8be4cd109d26f31c7` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:49.020660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] RBAC: Authorizing `identity:delete_project(project_id=67b547a04efd48c8be4cd109d26f31c7)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:49.023655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:49.165208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67b547a04efd48c8be4cd109d26f31c7', 'request_id': 'req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.181296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67b547a04efd48c8be4cd109d26f31c7', 'request_id': 'req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.203015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67b547a04efd48c8be4cd109d26f31c7', 'request_id': 'req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.231881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] The token cache is being invalidate because project 67b547a04efd48c8be4cd109d26f31c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:49.232163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.232850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.234737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1152/2277] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:48 2022] DELETE /identity/v3/projects/67b547a04efd48c8be4cd109d26f31c7 => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:49.240600 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:49.260666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c4c0f3-f34e-4d50-9ba1-4afb09724b82 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:49.397406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:49.398072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:49.398344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:49.398610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] PATH_INFO: `/v3/users/15c3b558609a4432bde2ead23d1ca690` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:49.399892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=15c3b558609a4432bde2ead23d1ca690)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:49.402120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:49.461101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15c3b558609a4432bde2ead23d1ca690'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.470338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15c3b558609a4432bde2ead23d1ca690'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.480802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15c3b558609a4432bde2ead23d1ca690'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.481493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15c3b558609a4432bde2ead23d1ca690'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.482141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-4d79615a-ff1c-440c-9c57-b846737830b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15c3b558609a4432bde2ead23d1ca690'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:49.514477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1126/2278] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:24:49 2022] DELETE /identity/v3/users/15c3b558609a4432bde2ead23d1ca690 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:49.876285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:49.878645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:49.893874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b0880-fff8-4bc7-9585-90e9aa37d09c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:49.982670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:49.983448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:49.983728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:49.984001 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] PATH_INFO: `/v3/projects/34d36583230e4b7585657541e355efdd` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:49.985159 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] RBAC: Authorizing `identity:delete_project(project_id=34d36583230e4b7585657541e355efdd)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:49.987236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:50.069981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34d36583230e4b7585657541e355efdd', 'request_id': 'req-bbdd3b60-1104-42cb-93ab-5c5842e9faec'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:50.078848 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34d36583230e4b7585657541e355efdd', 'request_id': 'req-bbdd3b60-1104-42cb-93ab-5c5842e9faec'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:50.087553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:50.091104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34d36583230e4b7585657541e355efdd', 'request_id': 'req-bbdd3b60-1104-42cb-93ab-5c5842e9faec'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:50.114665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] The token cache is being invalidate because project 34d36583230e4b7585657541e355efdd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:24:50.115016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:50.116018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bbdd3b60-1104-42cb-93ab-5c5842e9faec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:24:50.120896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1153/2279] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:24:49 2022] DELETE /identity/v3/projects/34d36583230e4b7585657541e355efdd => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:24:50.122875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12951475-c0a4-478e-acc9-a3b50fa4f914 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.123199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12951475-c0a4-478e-acc9-a3b50fa4f914 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.123586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12951475-c0a4-478e-acc9-a3b50fa4f914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.171382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-12951475-c0a4-478e-acc9-a3b50fa4f914 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:50.174127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-12951475-c0a4-478e-acc9-a3b50fa4f914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:50.198937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:50.307275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1127/2280] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:50.316971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad2158f-06c8-4bc0-aeff-7e290b8ae5d2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.317510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad2158f-06c8-4bc0-aeff-7e290b8ae5d2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.317852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad2158f-06c8-4bc0-aeff-7e290b8ae5d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.351213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-aad2158f-06c8-4bc0-aeff-7e290b8ae5d2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:50.353711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-aad2158f-06c8-4bc0-aeff-7e290b8ae5d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:50.363606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1154/2281] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:50.367970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12951475-c0a4-478e-acc9-a3b50fa4f914 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:50.378416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12951475-c0a4-478e-acc9-a3b50fa4f914 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:50.385177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:50.386091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.386659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.386934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.387603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:24:50.387908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:50.389851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:50.425915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1128/2282] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:24:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:24:50.430501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad2158f-06c8-4bc0-aeff-7e290b8ae5d2 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:50.442523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad2158f-06c8-4bc0-aeff-7e290b8ae5d2 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:50.448229 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:50.448842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.449528 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.449906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.450546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:50.452439 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:50.501791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1155/2283] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:24:50 2022] POST /identity/v3/projects => generated 371 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:50.503143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:50.520465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9724bdf-54eb-4315-a2b4-88d7e96e503d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:50.639494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:50.648090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.648650 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.649512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.649763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:50.651763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:50.696681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1129/2284] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:24:50 2022] POST /identity/v3/users => generated 286 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:24:50.701624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:50.731162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6dbcfd-a6cc-42d1-ae23-6622ee57d02b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:50.740067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:50.740688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.740957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.741255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.741901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:50.744297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:50.760098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1156/2285] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:50 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:50.761253 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:50.775479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b67e3ee-d91d-44d8-99a8-c55e546ef818 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:50.788369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07953b6d-82db-4025-a251-ccc1a6968287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:50.789170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07953b6d-82db-4025-a251-ccc1a6968287 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.789321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07953b6d-82db-4025-a251-ccc1a6968287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.789587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07953b6d-82db-4025-a251-ccc1a6968287 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.790255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07953b6d-82db-4025-a251-ccc1a6968287 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:50.792002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07953b6d-82db-4025-a251-ccc1a6968287 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:50.805167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1130/2286] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:24:50 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:24:50.808909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:50.819427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21b4fce-a042-4d1d-a833-d471abbb1a43 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:50.824211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:50.825396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.825838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.826116 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] PATH_INFO: `/v3/projects/7cb3e3496c1a45c8942c3678c0c299e5/users/2e60cf2931f04f0e8e911adea32979dd/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.842540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cb3e3496c1a45c8942c3678c0c299e5, user_id=2e60cf2931f04f0e8e911adea32979dd, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:50.845216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:50.875892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1157/2287] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:24:50 2022] PUT /identity/v3/projects/7cb3e3496c1a45c8942c3678c0c299e5/users/2e60cf2931f04f0e8e911adea32979dd/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:24:50.878438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d1174-be52-4f9a-856d-70efe75b334a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:50.878837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d1174-be52-4f9a-856d-70efe75b334a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:50.879236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d1174-be52-4f9a-856d-70efe75b334a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:50.915985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e98d1174-be52-4f9a-856d-70efe75b334a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:50.920550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e98d1174-be52-4f9a-856d-70efe75b334a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:50.996828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1131/2288] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:24:51.006397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:51.009250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:51.104989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9903481-246d-4db4-9444-ebc95290f374 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:51.109251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:51.110632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:51.111022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:51.111418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:51.132294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:51.138102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:51.155810 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1158/2289] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:51.906426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4349f-63e0-489f-9882-55d7ce956d99 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:51.907380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4349f-63e0-489f-9882-55d7ce956d99 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:51.907789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4349f-63e0-489f-9882-55d7ce956d99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:51.947723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-18c4349f-63e0-489f-9882-55d7ce956d99 None None] MFA Rules not processed for user `399110b53734477c98318822e79eb390`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:51.949047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-18c4349f-63e0-489f-9882-55d7ce956d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:51.969012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:52.092213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1132/2290] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:51 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:52.108614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:52.111011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:52.265218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72776f2f-2274-440c-9f49-72ddfcd06a83 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:52.278535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:52.279421 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:52.279892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:52.280163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:52.293785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60190554-0c00-4199-afd8-cbc47ac7b327 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:52.294032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:52.294147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60190554-0c00-4199-afd8-cbc47ac7b327 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:52.294394 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60190554-0c00-4199-afd8-cbc47ac7b327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:52.330625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:24:52.333703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-60190554-0c00-4199-afd8-cbc47ac7b327 None None] MFA Rules not processed for user `2e60cf2931f04f0e8e911adea32979dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:52.334944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-60190554-0c00-4199-afd8-cbc47ac7b327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:52.411463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:52.416953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:52.429966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1159/2291] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:52 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:24:52.434041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1133/2292] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:52 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:52.442034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:24:52.443745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:52.523521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51bf0e6-a50e-4d18-bff7-f416e1277305 service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:24:52.528640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214fba9c-368d-4e24-a5ff-de25acf95ff1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:24:52.529152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fba9c-368d-4e24-a5ff-de25acf95ff1 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:52.529457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fba9c-368d-4e24-a5ff-de25acf95ff1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:52.529727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fba9c-368d-4e24-a5ff-de25acf95ff1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:52.542293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214fba9c-368d-4e24-a5ff-de25acf95ff1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:24:52.544124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214fba9c-368d-4e24-a5ff-de25acf95ff1 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:24:52.556568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1160/2293] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:24:52 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:24:54.000623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f30073d-91c5-4941-9b06-773724c416db None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:54.001263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f30073d-91c5-4941-9b06-773724c416db None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:54.001263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f30073d-91c5-4941-9b06-773724c416db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:54.036487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-6f30073d-91c5-4941-9b06-773724c416db None None] MFA Rules not processed for user `642291c4daa04a6ab3fbce92a8be39c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:54.037619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6f30073d-91c5-4941-9b06-773724c416db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:54.049800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1134/2294] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:53 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:24:54.093307 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84da28e-fc29-419d-9e41-a4725e950d29 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:24:54.094007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84da28e-fc29-419d-9e41-a4725e950d29 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:24:54.094007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84da28e-fc29-419d-9e41-a4725e950d29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:24:54.116378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b84da28e-fc29-419d-9e41-a4725e950d29 None None] MFA Rules not processed for user `2e60cf2931f04f0e8e911adea32979dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:24:54.117638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b84da28e-fc29-419d-9e41-a4725e950d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:24:54.126823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1161/2295] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:24:54 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:04.031183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=57 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:04.109747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=46 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:18.419091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f30073d-91c5-4941-9b06-773724c416db None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:18.420530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6f30073d-91c5-4941-9b06-773724c416db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:18.434223 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f30073d-91c5-4941-9b06-773724c416db None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:18.438638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:18.439273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:18.439545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:18.439805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] PATH_INFO: `/v3/users/05a8c13e33e44c6585424325f7420fee` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:18.440904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] RBAC: Authorizing `identity:delete_user(user_id=05a8c13e33e44c6585424325f7420fee)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:18.443166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:18.494251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05a8c13e33e44c6585424325f7420fee'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:18.501787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05a8c13e33e44c6585424325f7420fee'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:18.512319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05a8c13e33e44c6585424325f7420fee'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:18.513062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05a8c13e33e44c6585424325f7420fee'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:18.514025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05a8c13e33e44c6585424325f7420fee'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:18.535678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1135/2296] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:25:18 2022] DELETE /identity/v3/users/05a8c13e33e44c6585424325f7420fee => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:18.540018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:18.540487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:18.540678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:18.561522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:18.562921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:18.636531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1162/2297] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:18.643683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:18.646006 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:18.718835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6115b56-ce08-48eb-b353-209bd268f17c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:18.723151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c373e0-a764-4e93-b500-d5aed32d87a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:18.723781 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c373e0-a764-4e93-b500-d5aed32d87a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:18.724059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c373e0-a764-4e93-b500-d5aed32d87a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:18.724411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c373e0-a764-4e93-b500-d5aed32d87a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:18.734167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c373e0-a764-4e93-b500-d5aed32d87a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:18.735581 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c373e0-a764-4e93-b500-d5aed32d87a6 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:18.745708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1136/2298] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:19.111333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.115187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:19.139030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f56ef64-8507-4722-ad64-4912de66af8c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.144340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.145964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.146601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.146968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] PATH_INFO: `/v3/projects/a895b94351214ca6a67e82cb34915530` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.148397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a895b94351214ca6a67e82cb34915530)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.150448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.230022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a895b94351214ca6a67e82cb34915530', 'request_id': 'req-124668e6-438a-4373-877a-22fac9b8b46f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:19.238060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a895b94351214ca6a67e82cb34915530', 'request_id': 'req-124668e6-438a-4373-877a-22fac9b8b46f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:19.245213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a895b94351214ca6a67e82cb34915530', 'request_id': 'req-124668e6-438a-4373-877a-22fac9b8b46f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:19.260472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] The token cache is being invalidate because project a895b94351214ca6a67e82cb34915530 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:25:19.260893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:19.262301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-124668e6-438a-4373-877a-22fac9b8b46f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:19.264370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1163/2299] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:25:19 2022] DELETE /identity/v3/projects/a895b94351214ca6a67e82cb34915530 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:19.269080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabbce6-aafb-40f1-9035-4c22a863dbbc None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.269588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabbce6-aafb-40f1-9035-4c22a863dbbc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.269853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabbce6-aafb-40f1-9035-4c22a863dbbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.290993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ceabbce6-aafb-40f1-9035-4c22a863dbbc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:19.292212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ceabbce6-aafb-40f1-9035-4c22a863dbbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:19.355954 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1137/2300] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:19.365177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c4176f-764b-410f-bb85-146919ba6016 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.365461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c4176f-764b-410f-bb85-146919ba6016 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.365728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c4176f-764b-410f-bb85-146919ba6016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.387616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-84c4176f-764b-410f-bb85-146919ba6016 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:19.388815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-84c4176f-764b-410f-bb85-146919ba6016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:19.398751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1164/2301] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:19.403220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabbce6-aafb-40f1-9035-4c22a863dbbc None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.413692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabbce6-aafb-40f1-9035-4c22a863dbbc None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.418558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.419112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.419652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.419652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.420304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:25:19.420608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.422427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.434622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1138/2302] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:25:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:25:19.439199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c4176f-764b-410f-bb85-146919ba6016 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.448956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c4176f-764b-410f-bb85-146919ba6016 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.454961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.455542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.455817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.456226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.456979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.458951 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.490108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1165/2303] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:25:19 2022] POST /identity/v3/projects => generated 369 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:25:19.494532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.499969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:19.507032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa3f00f-bc79-4bed-9ea1-db8f5940f0e8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.571287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.571893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.572165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.572430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.573072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.575333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.596822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1139/2304] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:25:19 2022] POST /identity/v3/users => generated 285 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:25:19.602543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.614585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ca2cf3-82e1-45e4-801a-17f1c2ae5faf admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.620351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.620962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.621258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.621533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.622166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.623757 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.636153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1166/2305] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:25:19.641300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.653278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d41051-cced-4542-8cad-98ca6883f53e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.658803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4606364d-bafe-4a70-a08c-1f3eac10625a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.659414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606364d-bafe-4a70-a08c-1f3eac10625a admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.659712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606364d-bafe-4a70-a08c-1f3eac10625a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.660470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606364d-bafe-4a70-a08c-1f3eac10625a admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.661162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4606364d-bafe-4a70-a08c-1f3eac10625a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.662804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4606364d-bafe-4a70-a08c-1f3eac10625a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.674627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1140/2306] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:25:19.679566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.689776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0938c04f-f844-4373-805b-cf01c25818f3 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.695199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.695851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.696160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.696399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] PATH_INFO: `/v3/projects/fc0c4ef439c541579a79b7e90e282938/users/cd303e6491b04603a39557b5745af9d2/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.714408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc0c4ef439c541579a79b7e90e282938, user_id=cd303e6491b04603a39557b5745af9d2, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.718518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.735319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1167/2307] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:25:19 2022] PUT /identity/v3/projects/fc0c4ef439c541579a79b7e90e282938/users/cd303e6491b04603a39557b5745af9d2/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:19.746526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a44b684-505a-4f5e-b81d-232f556f0d45 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.746891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a44b684-505a-4f5e-b81d-232f556f0d45 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.747363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a44b684-505a-4f5e-b81d-232f556f0d45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.786170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2a44b684-505a-4f5e-b81d-232f556f0d45 None None] MFA Rules not processed for user `cd303e6491b04603a39557b5745af9d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:19.787428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2a44b684-505a-4f5e-b81d-232f556f0d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:19.854940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1141/2308] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:19 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:19.867178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:19.868847 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:19.892326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:19.933192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270a4c88-e5ed-444d-ae24-761511167416 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:19.936990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb342a9-1894-4047-8c40-c280e76bc8b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:19.937643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb342a9-1894-4047-8c40-c280e76bc8b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:19.937919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb342a9-1894-4047-8c40-c280e76bc8b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:19.938192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb342a9-1894-4047-8c40-c280e76bc8b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:19.946758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb342a9-1894-4047-8c40-c280e76bc8b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:19.948282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb342a9-1894-4047-8c40-c280e76bc8b4 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:19.958034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1168/2309] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:19 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:19.991979 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a44b684-505a-4f5e-b81d-232f556f0d45 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:20.059758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a44b684-505a-4f5e-b81d-232f556f0d45 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:20.064260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:20.064962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:20.065284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:20.065847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] PATH_INFO: `/v3/users/cd303e6491b04603a39557b5745af9d2` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:20.067061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd303e6491b04603a39557b5745af9d2)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:20.068975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:20.083877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:20.124760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd303e6491b04603a39557b5745af9d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.132307 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd303e6491b04603a39557b5745af9d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.140977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd303e6491b04603a39557b5745af9d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.142296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd303e6491b04603a39557b5745af9d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.143014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd303e6491b04603a39557b5745af9d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.163583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1142/2310] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:25:19 2022] DELETE /identity/v3/users/cd303e6491b04603a39557b5745af9d2 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:20.167885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:20.168164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:20.168432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:20.189349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:20.190918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:20.257040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1169/2311] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:20.266918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:20.268351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:20.324357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d924e-654a-4de2-8f82-25461cf425b1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:20.329329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a88c503-36de-48cc-b5a0-4ee10ff881dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:20.329961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88c503-36de-48cc-b5a0-4ee10ff881dc service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:20.330231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88c503-36de-48cc-b5a0-4ee10ff881dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:20.330496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88c503-36de-48cc-b5a0-4ee10ff881dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:20.340613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a88c503-36de-48cc-b5a0-4ee10ff881dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:20.342234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a88c503-36de-48cc-b5a0-4ee10ff881dc service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:20.353244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1143/2312] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:20.738242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:20.739902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:20.752556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab5a78e-4a48-4f36-a659-80c7e19732f9 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:20.758800 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:20.759396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:20.759701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:20.760178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] PATH_INFO: `/v3/projects/fc0c4ef439c541579a79b7e90e282938` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:20.761474 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc0c4ef439c541579a79b7e90e282938)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:20.763440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:20.817866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc0c4ef439c541579a79b7e90e282938', 'request_id': 'req-40855716-5f37-47b9-b62a-d52c648e76c5'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.823905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc0c4ef439c541579a79b7e90e282938', 'request_id': 'req-40855716-5f37-47b9-b62a-d52c648e76c5'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.830112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc0c4ef439c541579a79b7e90e282938', 'request_id': 'req-40855716-5f37-47b9-b62a-d52c648e76c5'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.843304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] The token cache is being invalidate because project fc0c4ef439c541579a79b7e90e282938 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:25:20.843629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.844260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-40855716-5f37-47b9-b62a-d52c648e76c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:20.845955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1170/2313] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:25:20 2022] DELETE /identity/v3/projects/fc0c4ef439c541579a79b7e90e282938 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:20.851487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164214f-b453-4cd4-806f-1a58d2196ff0 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:20.852019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164214f-b453-4cd4-806f-1a58d2196ff0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:20.852197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164214f-b453-4cd4-806f-1a58d2196ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:20.878104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5164214f-b453-4cd4-806f-1a58d2196ff0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:20.880041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5164214f-b453-4cd4-806f-1a58d2196ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:20.942422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1144/2314] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:20.950806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371c051c-9248-4715-8f8b-1fe2c04dc991 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:20.951480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371c051c-9248-4715-8f8b-1fe2c04dc991 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:20.951723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371c051c-9248-4715-8f8b-1fe2c04dc991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:20.973811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-371c051c-9248-4715-8f8b-1fe2c04dc991 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:20.975110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-371c051c-9248-4715-8f8b-1fe2c04dc991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:20.987723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1171/2315] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:20.992297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5164214f-b453-4cd4-806f-1a58d2196ff0 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:21.002130 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5164214f-b453-4cd4-806f-1a58d2196ff0 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:21.007025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:21.007641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.007901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.008184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.008911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:25:21.009184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:21.010820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:21.025400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1145/2316] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:25:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:25:21.028220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371c051c-9248-4715-8f8b-1fe2c04dc991 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:21.039802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371c051c-9248-4715-8f8b-1fe2c04dc991 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:21.047958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:21.048577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.048850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.049113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.049823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:21.051937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:21.089486 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1172/2317] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:25:21 2022] POST /identity/v3/projects => generated 373 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:25:21.090896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:21.102457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3c51c2-6151-46a8-9e00-5978687505e3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:21.125633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:21.172585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:21.173388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.173657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.173917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.174657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:21.177317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:21.201058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1146/2318] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:25:21 2022] POST /identity/v3/users => generated 287 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:25:21.205296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:21.217964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ccb746-2793-4267-979d-e6a959086fde admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:21.223793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:21.224443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.224699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.224965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.225645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:21.227392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:21.243908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1173/2319] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:25:21.246136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:21.257246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333c0ff8-9da0-4620-b0fc-61cd5031ae08 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:21.263388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84d54db-8cca-4242-aec6-409a6928e0f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:21.263935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d54db-8cca-4242-aec6-409a6928e0f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.264417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d54db-8cca-4242-aec6-409a6928e0f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.264594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d54db-8cca-4242-aec6-409a6928e0f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.265286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84d54db-8cca-4242-aec6-409a6928e0f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:21.266862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84d54db-8cca-4242-aec6-409a6928e0f8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:21.281205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1147/2320] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:25:21.285718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:21.300745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b763f55a-4b21-48b2-8eb6-27debd855c26 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:21.306045 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:21.306705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.306981 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.307404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] PATH_INFO: `/v3/projects/fdf11a2660f74e3999f378f2fc427e87/users/9d91281032a64975a575b4a213118327/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.309840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:21.328632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdf11a2660f74e3999f378f2fc427e87, user_id=9d91281032a64975a575b4a213118327, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:21.331441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:21.345769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1174/2321] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:25:21 2022] PUT /identity/v3/projects/fdf11a2660f74e3999f378f2fc427e87/users/9d91281032a64975a575b4a213118327/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:21.357342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.357626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.357916 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.387016 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] MFA Rules not processed for user `9d91281032a64975a575b4a213118327`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:21.388235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:21.461007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1148/2322] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:21.471840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:21.474019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:21.532214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acbb81c-4af1-46da-b46a-3ccf1aee41d1 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:21.536169 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dbd411-969e-44da-b30c-09333bd98ec7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:21.536779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dbd411-969e-44da-b30c-09333bd98ec7 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:21.537052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dbd411-969e-44da-b30c-09333bd98ec7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:21.537342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dbd411-969e-44da-b30c-09333bd98ec7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:21.546758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dbd411-969e-44da-b30c-09333bd98ec7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:21.548193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dbd411-969e-44da-b30c-09333bd98ec7 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:21.569473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1175/2323] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:23.087525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:23.089258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:23.156407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bf816-2cbc-4e2d-b105-0afdfd019e5a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:23.162030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:23.162662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:23.162930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:23.163191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] PATH_INFO: `/v3/users/ed17451121384ab1b8aa47a88ad9c5ae` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:23.164438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed17451121384ab1b8aa47a88ad9c5ae)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:23.166530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:23.203603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:23.222576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed17451121384ab1b8aa47a88ad9c5ae'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.230657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed17451121384ab1b8aa47a88ad9c5ae'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.240451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed17451121384ab1b8aa47a88ad9c5ae'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.241099 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed17451121384ab1b8aa47a88ad9c5ae'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.241824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed17451121384ab1b8aa47a88ad9c5ae'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.260587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1149/2324] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:25:23 2022] DELETE /identity/v3/users/ed17451121384ab1b8aa47a88ad9c5ae => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:23.265205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:23.265684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:23.265786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:23.284632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:23.285892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:23.352137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1176/2325] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:23.361982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:23.364610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:23.438889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f1c85-7ec6-49fa-aa41-87bba59ec253 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:23.443028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3007cc8c-9a33-4821-8b0c-f9a5af1982b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:23.443666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3007cc8c-9a33-4821-8b0c-f9a5af1982b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:23.444015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3007cc8c-9a33-4821-8b0c-f9a5af1982b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:23.444344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3007cc8c-9a33-4821-8b0c-f9a5af1982b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:23.454102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3007cc8c-9a33-4821-8b0c-f9a5af1982b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:23.455610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3007cc8c-9a33-4821-8b0c-f9a5af1982b5 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:23.466452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1150/2326] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:23.835927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:23.837394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:23.850279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e86db2-7750-4aa9-8463-c078efe27265 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:23.854849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:23.855501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:23.856063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:23.856063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] PATH_INFO: `/v3/projects/faac909270b6414eabe3a4b144539f4d` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:23.857515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] RBAC: Authorizing `identity:delete_project(project_id=faac909270b6414eabe3a4b144539f4d)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:23.859416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:23.913007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'faac909270b6414eabe3a4b144539f4d', 'request_id': 'req-258a9086-007c-4554-b935-1d63ddfdaf66'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.920524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'faac909270b6414eabe3a4b144539f4d', 'request_id': 'req-258a9086-007c-4554-b935-1d63ddfdaf66'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.926602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'faac909270b6414eabe3a4b144539f4d', 'request_id': 'req-258a9086-007c-4554-b935-1d63ddfdaf66'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.948480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] The token cache is being invalidate because project faac909270b6414eabe3a4b144539f4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:25:23.949038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.950500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-258a9086-007c-4554-b935-1d63ddfdaf66 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:23.951975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1177/2327] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:25:23 2022] DELETE /identity/v3/projects/faac909270b6414eabe3a4b144539f4d => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:23.957172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2222701-1803-4ea0-96a6-8476c29319b9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:23.957644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2222701-1803-4ea0-96a6-8476c29319b9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:23.957780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2222701-1803-4ea0-96a6-8476c29319b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:23.991434 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-a2222701-1803-4ea0-96a6-8476c29319b9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:23.992739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a2222701-1803-4ea0-96a6-8476c29319b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:24.058782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1151/2328] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:24.066836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8666682-e07d-4693-8f0a-905cf9f010ac None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.067126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8666682-e07d-4693-8f0a-905cf9f010ac None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.068142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8666682-e07d-4693-8f0a-905cf9f010ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.093413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c8666682-e07d-4693-8f0a-905cf9f010ac None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:24.094640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c8666682-e07d-4693-8f0a-905cf9f010ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:24.103540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1178/2329] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:24.107939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2222701-1803-4ea0-96a6-8476c29319b9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:24.119100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2222701-1803-4ea0-96a6-8476c29319b9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:24.124878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:24.125532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.125802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.126066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.126732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:25:24.127032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:24.129157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:24.145133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1152/2330] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:25:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:25:24.153470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8666682-e07d-4693-8f0a-905cf9f010ac None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:24.159689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:24.164290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8666682-e07d-4693-8f0a-905cf9f010ac None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:24.169854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:24.170481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.171599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.171902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.172751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:24.174795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:24.209269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1179/2331] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:25:24 2022] POST /identity/v3/projects => generated 399 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:25:24.213231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:24.225289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe86fc-a5b8-4337-8993-321747c19a4a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:24.262203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:24.290997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:24.291605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.291872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.292172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.292763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:24.294727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:24.324074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1153/2332] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:25:24 2022] POST /identity/v3/users => generated 300 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:25:24.328893 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:24.341737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259280ad-c353-4a4d-a960-16b3e074f4ba admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:24.347084 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:24.347808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.348628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.348628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.349169 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:24.351024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:24.362785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1180/2333] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:24 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:25:24.366933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:24.378624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6020cb77-ea40-4f00-87b4-452711902bd1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:24.384198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d6baf6-8b52-4dc4-b786-35be23361749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:24.384837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d6baf6-8b52-4dc4-b786-35be23361749 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.385155 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d6baf6-8b52-4dc4-b786-35be23361749 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.385534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d6baf6-8b52-4dc4-b786-35be23361749 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.386207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d6baf6-8b52-4dc4-b786-35be23361749 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:24.389583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d6baf6-8b52-4dc4-b786-35be23361749 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:24.401189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1154/2334] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:24 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:25:24.405428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:24.421012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29825e0d-890e-4aef-9a30-7f32aadb934a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:24.428269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:24.429012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.429369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.429826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] PATH_INFO: `/v3/projects/35ed6547d15841428a00838658626c57/users/71a20c90321340198f2c0251f389736c/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.450898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35ed6547d15841428a00838658626c57, user_id=71a20c90321340198f2c0251f389736c, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:24.453914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:24.486254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1181/2335] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:25:24 2022] PUT /identity/v3/projects/35ed6547d15841428a00838658626c57/users/71a20c90321340198f2c0251f389736c/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:25:24.499178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec421145-3d42-445f-847b-5f7c9b41a28b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.499414 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec421145-3d42-445f-847b-5f7c9b41a28b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.499689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec421145-3d42-445f-847b-5f7c9b41a28b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.567647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ec421145-3d42-445f-847b-5f7c9b41a28b None None] MFA Rules not processed for user `71a20c90321340198f2c0251f389736c`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:24.569868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ec421145-3d42-445f-847b-5f7c9b41a28b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:24.676304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1155/2336] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:24 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:24.695096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:24.697588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:24.765350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4b619c-f072-4c85-ab38-4a9f31f2a072 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:24.769083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e61880f-a462-4c7f-9330-44be346922b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:24.769878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61880f-a462-4c7f-9330-44be346922b2 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:24.770215 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61880f-a462-4c7f-9330-44be346922b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:24.770551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61880f-a462-4c7f-9330-44be346922b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:24.780431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e61880f-a462-4c7f-9330-44be346922b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:24.781908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e61880f-a462-4c7f-9330-44be346922b2 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:24.791056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1182/2337] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:24 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:34.670774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=49,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:34.786969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=49,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:36.144542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec421145-3d42-445f-847b-5f7c9b41a28b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:36.254072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec421145-3d42-445f-847b-5f7c9b41a28b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:36.260209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:36.261274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:36.261797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:36.262249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] PATH_INFO: `/v3/users/71a20c90321340198f2c0251f389736c` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:36.263618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] RBAC: Authorizing `identity:delete_user(user_id=71a20c90321340198f2c0251f389736c)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:36.266610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:36.333697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71a20c90321340198f2c0251f389736c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.341660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71a20c90321340198f2c0251f389736c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.353664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71a20c90321340198f2c0251f389736c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.354379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71a20c90321340198f2c0251f389736c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.355055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71a20c90321340198f2c0251f389736c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.384656 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1156/2338] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:25:36 2022] DELETE /identity/v3/users/71a20c90321340198f2c0251f389736c => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:36.386711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23eb031-879e-4277-bc79-886def85c17a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:36.386711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23eb031-879e-4277-bc79-886def85c17a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:36.387320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23eb031-879e-4277-bc79-886def85c17a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:36.413019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:36.414870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:36.415239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a23eb031-879e-4277-bc79-886def85c17a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:36.416478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a23eb031-879e-4277-bc79-886def85c17a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:36.493385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1183/2339] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:25:36.533672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0ee4f-2f10-4c3d-ba6b-8170e9af044b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:36.538385 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23eb031-879e-4277-bc79-886def85c17a None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:36.540300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a23eb031-879e-4277-bc79-886def85c17a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:36.541958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:36.542750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:36.543119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:36.543484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] PATH_INFO: `/v3/users/9d91281032a64975a575b4a213118327` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:36.545744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d91281032a64975a575b4a213118327)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:36.559275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:36.595888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:36.630279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d91281032a64975a575b4a213118327'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.632860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23eb031-879e-4277-bc79-886def85c17a None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:36.636599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:36.637292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:36.637442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d91281032a64975a575b4a213118327'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.637584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:36.637827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:36.646355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d91281032a64975a575b4a213118327'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.647015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d91281032a64975a575b4a213118327'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.647699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5fc28bb3-b4ee-4480-8097-e6565cdcc84b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d91281032a64975a575b4a213118327'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:36.650000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:36.651456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:36.662078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1184/2340] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:36.674486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1157/2341] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:25:36 2022] DELETE /identity/v3/users/9d91281032a64975a575b4a213118327 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:36.674678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:36.674678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:36.686837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:36.687484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:36.687989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:36.716308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:36.718251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:36.782955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077ca6fa-5d3c-4163-8298-d6a9d24432a6 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:36.812375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1158/2342] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:36.826514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:36.828416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:36.845044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:36.866784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:36.868574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:36.869011 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:36.869315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:36.871126 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:36.883928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:36.885104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:36.896643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1185/2343] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:25:36.903480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f88de7-e066-460c-8238-ffcc230b453c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:36.908818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9042479f-e861-4380-8798-c4692859209d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:36.909484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9042479f-e861-4380-8798-c4692859209d service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:36.909791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9042479f-e861-4380-8798-c4692859209d service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:36.910248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9042479f-e861-4380-8798-c4692859209d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:36.922131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9042479f-e861-4380-8798-c4692859209d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:36.923571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9042479f-e861-4380-8798-c4692859209d service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:36.935997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1159/2344] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:37.743444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:37.744962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:37.758632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fac7de-faa2-4f59-91af-dfedb65fa96c service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:37.763684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:37.764449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:37.764822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:37.765242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] PATH_INFO: `/v3/projects/35ed6547d15841428a00838658626c57` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:37.769596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=35ed6547d15841428a00838658626c57)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:37.771597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:37.809159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9042479f-e861-4380-8798-c4692859209d service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:37.810700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9042479f-e861-4380-8798-c4692859209d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:37.835069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9042479f-e861-4380-8798-c4692859209d service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:37.855058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35ed6547d15841428a00838658626c57', 'request_id': 'req-f830b7e5-e8e1-4059-8594-0d53774579d0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:37.863219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35ed6547d15841428a00838658626c57', 'request_id': 'req-f830b7e5-e8e1-4059-8594-0d53774579d0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:37.875972 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35ed6547d15841428a00838658626c57', 'request_id': 'req-f830b7e5-e8e1-4059-8594-0d53774579d0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:37.894437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] The token cache is being invalidate because project 35ed6547d15841428a00838658626c57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:25:37.895091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:37.895950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f830b7e5-e8e1-4059-8594-0d53774579d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:37.902309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:37.905638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1186/2345] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:25:37 2022] DELETE /identity/v3/projects/35ed6547d15841428a00838658626c57 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:37.927612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:37.928766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:37.929103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:37.929463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] PATH_INFO: `/v3/projects/fdf11a2660f74e3999f378f2fc427e87` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:37.930878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdf11a2660f74e3999f378f2fc427e87)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:37.933340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.004747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdf11a2660f74e3999f378f2fc427e87', 'request_id': 'req-33856ef5-4789-4a1b-a9aa-32d4293fd553'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:38.017547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdf11a2660f74e3999f378f2fc427e87', 'request_id': 'req-33856ef5-4789-4a1b-a9aa-32d4293fd553'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:38.025873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdf11a2660f74e3999f378f2fc427e87', 'request_id': 'req-33856ef5-4789-4a1b-a9aa-32d4293fd553'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:38.038390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] The token cache is being invalidate because project fdf11a2660f74e3999f378f2fc427e87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:25:38.038807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:38.039643 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-33856ef5-4789-4a1b-a9aa-32d4293fd553 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:25:38.045219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1160/2346] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:25:37 2022] DELETE /identity/v3/projects/fdf11a2660f74e3999f378f2fc427e87 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:38.048851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6356be-80f1-4d06-a2b6-c83d20a5c999 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.049233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6356be-80f1-4d06-a2b6-c83d20a5c999 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.049602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6356be-80f1-4d06-a2b6-c83d20a5c999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.077751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4e6356be-80f1-4d06-a2b6-c83d20a5c999 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:38.079279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4e6356be-80f1-4d06-a2b6-c83d20a5c999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:38.150032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1187/2347] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:38.158192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cf7cc8-ab7c-4285-96d8-7371f119e2b6 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.158431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cf7cc8-ab7c-4285-96d8-7371f119e2b6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.158707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cf7cc8-ab7c-4285-96d8-7371f119e2b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.181548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-60cf7cc8-ab7c-4285-96d8-7371f119e2b6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:38.183366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-60cf7cc8-ab7c-4285-96d8-7371f119e2b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:38.194250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1161/2348] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:38.199957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6356be-80f1-4d06-a2b6-c83d20a5c999 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:38.217097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6356be-80f1-4d06-a2b6-c83d20a5c999 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:38.225556 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:38.226156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.226443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.226708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.227362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:25:38.227661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:38.229287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.241531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1188/2349] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:25:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:25:38.246192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cf7cc8-ab7c-4285-96d8-7371f119e2b6 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:38.255858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cf7cc8-ab7c-4285-96d8-7371f119e2b6 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:38.260611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:38.261244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.261516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.261785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.262416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:38.264289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.294149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1162/2350] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:25:38 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:25:38.298628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:38.308879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5facd97a-80d4-4fd4-b107-dc2e1e59ad51 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:38.352073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:38.367809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:38.368424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.368699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.368974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.369636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:38.371553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.395064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1189/2351] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:25:38 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:25:38.399528 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:38.410986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d3470-b38c-4457-95ae-7dcef70c690d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:38.416039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:38.416640 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.416917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.417226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.417891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:38.419497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.430760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1163/2352] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:38 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:25:38.436705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:38.453614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823e833-42be-4560-8d60-02b2d828cb5c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:38.460129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cd945e-d9c6-4c6b-8e26-0591874fa274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:38.460734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cd945e-d9c6-4c6b-8e26-0591874fa274 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.461006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cd945e-d9c6-4c6b-8e26-0591874fa274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.461295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cd945e-d9c6-4c6b-8e26-0591874fa274 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.461935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cd945e-d9c6-4c6b-8e26-0591874fa274 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:38.463523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cd945e-d9c6-4c6b-8e26-0591874fa274 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.474911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1190/2353] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:25:38 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:25:38.481436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:38.494579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecc8ad1-687b-4409-8f5b-b69e9d9b3a0d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:38.500172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:38.500997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.501395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.501683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] PATH_INFO: `/v3/projects/fa43d54225bf44b283f9db5ddbbb13ca/users/75744eccc3d24d4c9282cbe3a3314489/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.520674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa43d54225bf44b283f9db5ddbbb13ca, user_id=75744eccc3d24d4c9282cbe3a3314489, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:38.523363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.539454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1164/2354] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:25:38 2022] PUT /identity/v3/projects/fa43d54225bf44b283f9db5ddbbb13ca/users/75744eccc3d24d4c9282cbe3a3314489/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:25:38.545515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415face0-d3d3-4a56-8707-74ba544d0b8e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.546148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415face0-d3d3-4a56-8707-74ba544d0b8e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.546586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415face0-d3d3-4a56-8707-74ba544d0b8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.568727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-415face0-d3d3-4a56-8707-74ba544d0b8e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:38.569978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-415face0-d3d3-4a56-8707-74ba544d0b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:38.632378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1191/2355] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:38.642730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:38.645308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:38.706481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aedc39-ec91-4c85-8f91-3646e24f5729 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:38.713437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:38.714380 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:38.714738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:38.714912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:38.725952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:38.727461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:38.737436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1165/2356] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:39.495508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340379c9-850f-4055-a39d-2aef3644ed8d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:39.496100 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340379c9-850f-4055-a39d-2aef3644ed8d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:39.496100 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340379c9-850f-4055-a39d-2aef3644ed8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:39.529303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-340379c9-850f-4055-a39d-2aef3644ed8d None None] MFA Rules not processed for user `75744eccc3d24d4c9282cbe3a3314489`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:39.530433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-340379c9-850f-4055-a39d-2aef3644ed8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:39.600906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1192/2357] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:39.611958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:25:39.613681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:39.627652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:39.684602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b2e02-5f8f-4d8b-a708-c01e9a592cfe service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:25:39.688243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39099c82-cb25-44e7-b8fa-27af0b0510ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:25:39.688849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39099c82-cb25-44e7-b8fa-27af0b0510ba service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:39.689105 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39099c82-cb25-44e7-b8fa-27af0b0510ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:39.689418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39099c82-cb25-44e7-b8fa-27af0b0510ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:39.698615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39099c82-cb25-44e7-b8fa-27af0b0510ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:25:39.700061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39099c82-cb25-44e7-b8fa-27af0b0510ba service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:25:39.713276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1166/2358] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:25:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:25:41.114625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd44ae3-dc50-4d11-a2d7-e29c930b3de5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:41.115210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd44ae3-dc50-4d11-a2d7-e29c930b3de5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:41.115210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd44ae3-dc50-4d11-a2d7-e29c930b3de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:41.135310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1cd44ae3-dc50-4d11-a2d7-e29c930b3de5 None None] MFA Rules not processed for user `75744eccc3d24d4c9282cbe3a3314489`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:41.136509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1cd44ae3-dc50-4d11-a2d7-e29c930b3de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:41.148608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1193/2359] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:43.274739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:43.275303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:43.275303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:43.294941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] MFA Rules not processed for user `2e60cf2931f04f0e8e911adea32979dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:43.296329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:43.369548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1167/2360] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:43 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:25:45.184896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb234c-e724-449a-8130-3a9d549daf3d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:25:45.185414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb234c-e724-449a-8130-3a9d549daf3d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:25:45.185509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb234c-e724-449a-8130-3a9d549daf3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:25:45.205370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-acbb234c-e724-449a-8130-3a9d549daf3d None None] MFA Rules not processed for user `d945d512ae034ec08838d0a3afde2429`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:25:45.206986 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-acbb234c-e724-449a-8130-3a9d549daf3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:25:45.256545 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:45.272821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1194/2361] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:25:45 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:25:53.360976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=46 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:25:55.264345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:26:02.360683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:02.362524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:02.379204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ab91cf-8eea-44a8-8be9-677f8581e35d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:02.383523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:02.384150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:02.384418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:02.384683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:02.386924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:02.462544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:02.463365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:02.473416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1168/2362] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:26:02 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:26:12.468750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=30 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:26:23.900419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d938e7-85c1-423b-8579-0a89eed3f1fe None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:23.900983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d938e7-85c1-423b-8579-0a89eed3f1fe None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:23.900983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d938e7-85c1-423b-8579-0a89eed3f1fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:23.921742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-35d938e7-85c1-423b-8579-0a89eed3f1fe None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:26:23.922952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-35d938e7-85c1-423b-8579-0a89eed3f1fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:23.987303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1195/2363] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:26:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:26:24.032063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:24.033766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:24.130080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd39e5d8-9390-4622-a79f-d447dddf34f1 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:24.136936 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:24.137645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:24.138078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:24.138503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:24.148219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:24.149926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:24.158629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1169/2364] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:26:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:26:33.979728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=25 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:26:34.155064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=32 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:26:41.230068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:41.230599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:41.230688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:41.252882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:26:41.254357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:41.266026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1196/2365] 10.222.0.52 () {58 vars in 967 bytes} [Sat Aug 27 02:26:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:26:41.280040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:41.297721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4302e98a-0ac5-4d05-b4b6-b05c047980b4 service swift] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:41.306994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:41.307791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:41.308207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:41.308597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:41.322529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:41.325385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:41.341279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1170/2366] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:26:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:26:43.544459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:43.546777 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:43.570038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8364b6-9dd6-48f0-8b6d-2a9c7e3317a1 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:43.575101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:43.575824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:43.576162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:43.576639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] PATH_INFO: `/v3/users/2e60cf2931f04f0e8e911adea32979dd` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:43.579029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e60cf2931f04f0e8e911adea32979dd)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:43.581312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:43.658668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e60cf2931f04f0e8e911adea32979dd'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:43.672140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e60cf2931f04f0e8e911adea32979dd'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:43.673021 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e60cf2931f04f0e8e911adea32979dd'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:43.686250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e60cf2931f04f0e8e911adea32979dd'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:43.696142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-acfba402-f033-4e3e-8648-193a87b83916 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e60cf2931f04f0e8e911adea32979dd'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:43.703940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1197/2367] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:26:43 2022] DELETE /identity/v3/users/2e60cf2931f04f0e8e911adea32979dd => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:26:43.894119 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:43.908470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de5bf77-3548-409c-a771-8c82d8002b47 service swift] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:43.974664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:43.975440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:43.975820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:43.976234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] PATH_INFO: `/v3/projects/7cb3e3496c1a45c8942c3678c0c299e5` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:43.977674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cb3e3496c1a45c8942c3678c0c299e5)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:43.980201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.034353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cb3e3496c1a45c8942c3678c0c299e5', 'request_id': 'req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:44.046798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cb3e3496c1a45c8942c3678c0c299e5', 'request_id': 'req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:44.055418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cb3e3496c1a45c8942c3678c0c299e5', 'request_id': 'req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:44.068287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] The token cache is being invalidate because project 7cb3e3496c1a45c8942c3678c0c299e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:26:44.068585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:44.071699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9e202cfe-ea2c-471c-b7a3-aff7d02e1e7d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:26:44.072882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6f587f-1d46-40a7-b6c4-664de2227dc9 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.073499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6f587f-1d46-40a7-b6c4-664de2227dc9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.073625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1171/2368] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:26:43 2022] DELETE /identity/v3/projects/7cb3e3496c1a45c8942c3678c0c299e5 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:26:44.073708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6f587f-1d46-40a7-b6c4-664de2227dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.078000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4127ec6-edee-4a3d-96e2-cc09cb8670bf None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.078279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4127ec6-edee-4a3d-96e2-cc09cb8670bf None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.078549 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4127ec6-edee-4a3d-96e2-cc09cb8670bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.096140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-9f6f587f-1d46-40a7-b6c4-664de2227dc9 None None] MFA Rules not processed for user `26a86351583d49d090b3db09f127d49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:26:44.098049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9f6f587f-1d46-40a7-b6c4-664de2227dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:44.103173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-e4127ec6-edee-4a3d-96e2-cc09cb8670bf None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:26:44.104927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e4127ec6-edee-4a3d-96e2-cc09cb8670bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:44.162608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1198/2369] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:26:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:26:44.185742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1172/2370] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:26:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:26:44.194259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c15fb3-9195-4463-95e9-3fbe09b147bb None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.194661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c15fb3-9195-4463-95e9-3fbe09b147bb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.195038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c15fb3-9195-4463-95e9-3fbe09b147bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.217743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-28c15fb3-9195-4463-95e9-3fbe09b147bb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:26:44.221255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-28c15fb3-9195-4463-95e9-3fbe09b147bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:44.230891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1199/2371] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:26:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:26:44.235175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4127ec6-edee-4a3d-96e2-cc09cb8670bf None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:44.246829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4127ec6-edee-4a3d-96e2-cc09cb8670bf None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:44.252299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:44.252989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.253353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.253618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.254895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:26:44.254895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:44.256727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.271188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1173/2372] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:26:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:26:44.276276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c15fb3-9195-4463-95e9-3fbe09b147bb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:44.288803 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c15fb3-9195-4463-95e9-3fbe09b147bb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:44.293447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:44.294064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.294332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.294619 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.295297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:44.297221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.331183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1200/2373] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:26:44 2022] POST /identity/v3/projects => generated 365 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:26:44.333379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:44.340156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:26:44.344326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9692680-afb8-4fa2-85ea-08936067d8ed admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:44.416646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:44.418328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.418647 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.418919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.419641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:44.422257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.449599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1174/2374] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:26:44 2022] POST /identity/v3/users => generated 283 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:26:44.450962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:44.463348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d132ae-1dbb-472d-a703-8c2c3ca1d343 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:44.467954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:44.468814 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.469096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.469408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.470060 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:44.471658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.483440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1201/2375] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:26:44.487776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:44.499539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b145b-3ffd-4052-b762-59c0d040f03f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:44.504263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc65b32-cda2-4b70-a0bf-6aceb7280d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:44.504861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc65b32-cda2-4b70-a0bf-6aceb7280d6b admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.505888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc65b32-cda2-4b70-a0bf-6aceb7280d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.506156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc65b32-cda2-4b70-a0bf-6aceb7280d6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.506811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc65b32-cda2-4b70-a0bf-6aceb7280d6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:44.508399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc65b32-cda2-4b70-a0bf-6aceb7280d6b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.521754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1175/2376] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:26:44.526109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:44.537504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4ad0a-55bd-4d48-99a2-335b8fa37bf0 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:44.543078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:44.544123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.544123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.544321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] PATH_INFO: `/v3/projects/602b1484f6cd4bad8190909a7d281255/users/493f498cc81d467d9d09654ada2e594f/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.561534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] RBAC: Authorizing `identity:create_grant(project_id=602b1484f6cd4bad8190909a7d281255, user_id=493f498cc81d467d9d09654ada2e594f, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:44.564365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.579674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1202/2377] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:26:44 2022] PUT /identity/v3/projects/602b1484f6cd4bad8190909a7d281255/users/493f498cc81d467d9d09654ada2e594f/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:26:44.590339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c581179-a890-4aa2-8d1b-931c1583ec92 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.590623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c581179-a890-4aa2-8d1b-931c1583ec92 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.590896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c581179-a890-4aa2-8d1b-931c1583ec92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.619605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4c581179-a890-4aa2-8d1b-931c1583ec92 None None] MFA Rules not processed for user `493f498cc81d467d9d09654ada2e594f`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:26:44.620225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4c581179-a890-4aa2-8d1b-931c1583ec92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:44.687685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1176/2378] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:26:44 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:26:44.699636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:26:44.701513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:26:44.719284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:26:44.774753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4420cce-77e1-4ad9-958a-05cd5d62decd admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:26:44.781845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:26:44.781845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:26:44.782185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:26:44.782348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:26:44.793351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:26:44.794918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:26:44.805050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1203/2379] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:26:44 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:26:54.684796 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=64,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:26:54.800892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=23 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:27:41.873444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:27:41.874048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:27:41.874048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:27:41.894010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] MFA Rules not processed for user `75744eccc3d24d4c9282cbe3a3314489`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:27:41.895217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:41.958641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1177/2380] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:27:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:27:42.001426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:27:42.003052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:42.064102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b4de3e-dddc-45cb-bfd0-a009cd9765fd service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:27:42.068097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:27:42.068708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:27:42.068990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:27:42.069287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:27:42.071192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:42.145887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:27:42.147333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:27:42.158838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1204/2381] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:27:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:27:42.370788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:27:42.372342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:42.434053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eab8ed1-02ca-47e0-b005-17dbcdb913b7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:27:42.438095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:27:42.438731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:27:42.439000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:27:42.439263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:27:42.441173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:42.451674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:27:42.453092 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:27:42.461605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1178/2382] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:27:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:27:42.800010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:27:42.801478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:42.815334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:27:42.824848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c85b2-1868-436e-ad7d-625c40deabb9 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:27:42.829745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:27:42.830485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:27:42.830765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:27:42.831033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:27:42.833239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:42.847914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:27:42.849631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:27:42.861143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1179/2383] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:27:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:27:42.886050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c886985-4f93-49ff-9eb7-c1f47e13c892 service glance] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:27:42.891325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:27:42.891943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:27:42.892219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:27:42.892493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:27:42.894316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:42.917364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:27:42.917364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:27:42.924104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1205/2384] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:27:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:27:52.856610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=57 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:27:52.919057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=86 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:27:54.691304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:27:54.693839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:54.705712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106f069b-25a2-408f-9c20-f5b8d99e753e service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:27:54.711390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2ef492-2954-4797-bdd7-a064cbc16a5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:27:54.712280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ef492-2954-4797-bdd7-a064cbc16a5b service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:27:54.713297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ef492-2954-4797-bdd7-a064cbc16a5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:27:54.713297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ef492-2954-4797-bdd7-a064cbc16a5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:27:54.715166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-de2ef492-2954-4797-bdd7-a064cbc16a5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:27:54.727004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2ef492-2954-4797-bdd7-a064cbc16a5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:27:54.729142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2ef492-2954-4797-bdd7-a064cbc16a5b service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:27:54.739608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1180/2385] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:27:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:28:00.871132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:00.943121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea792b5-5771-4caa-b390-cd2b60220a4a service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:00.959673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:00.960769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:00.961325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:00.961763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] PATH_INFO: `/v3/users/493f498cc81d467d9d09654ada2e594f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:00.963670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] RBAC: Authorizing `identity:delete_user(user_id=493f498cc81d467d9d09654ada2e594f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:00.966698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:01.034954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '493f498cc81d467d9d09654ada2e594f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.057871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '493f498cc81d467d9d09654ada2e594f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.058878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '493f498cc81d467d9d09654ada2e594f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.069213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '493f498cc81d467d9d09654ada2e594f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.076797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '493f498cc81d467d9d09654ada2e594f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.084195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1206/2386] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:28:00 2022] DELETE /identity/v3/users/493f498cc81d467d9d09654ada2e594f => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:28:01.089492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:01.090019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:01.090138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:01.110907 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:01.112134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:01.178740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1181/2387] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:28:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:01.188801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:01.190292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:01.259007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf9961c-8ff1-49b5-a0d5-278b4e9aff76 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:01.262947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a0ee59-a2b2-4dde-9e90-f63713c281c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:01.263559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0ee59-a2b2-4dde-9e90-f63713c281c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:01.263822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0ee59-a2b2-4dde-9e90-f63713c281c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:01.264091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0ee59-a2b2-4dde-9e90-f63713c281c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:01.274504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a0ee59-a2b2-4dde-9e90-f63713c281c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:01.275967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a0ee59-a2b2-4dde-9e90-f63713c281c8 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:01.287598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1207/2388] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:28:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:28:01.597428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:01.599218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:01.612334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fab963-e39d-4490-8cbf-d71780792685 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:01.617906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:01.618554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:01.618823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:01.619089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] PATH_INFO: `/v3/projects/602b1484f6cd4bad8190909a7d281255` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:01.620213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] RBAC: Authorizing `identity:delete_project(project_id=602b1484f6cd4bad8190909a7d281255)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:01.622311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:01.676329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '602b1484f6cd4bad8190909a7d281255', 'request_id': 'req-3a478b0e-2660-4d17-a43c-76ac3c92bc64'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.685896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '602b1484f6cd4bad8190909a7d281255', 'request_id': 'req-3a478b0e-2660-4d17-a43c-76ac3c92bc64'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.693359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '602b1484f6cd4bad8190909a7d281255', 'request_id': 'req-3a478b0e-2660-4d17-a43c-76ac3c92bc64'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.705072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] The token cache is being invalidate because project 602b1484f6cd4bad8190909a7d281255 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:28:01.705692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.706343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3a478b0e-2660-4d17-a43c-76ac3c92bc64 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:01.707845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1182/2389] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:28:01 2022] DELETE /identity/v3/projects/602b1484f6cd4bad8190909a7d281255 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:28:01.713091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eb0611-58a9-400c-959b-aafc3af2b960 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:01.713504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eb0611-58a9-400c-959b-aafc3af2b960 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:01.713837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eb0611-58a9-400c-959b-aafc3af2b960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:01.735300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-47eb0611-58a9-400c-959b-aafc3af2b960 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:01.736507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-47eb0611-58a9-400c-959b-aafc3af2b960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:01.754641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:01.807485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1208/2390] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:28:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:28:01.819878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4d90b5-8fbe-42c5-9111-5ccaaed7040c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:01.820619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4d90b5-8fbe-42c5-9111-5ccaaed7040c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:01.822011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4d90b5-8fbe-42c5-9111-5ccaaed7040c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:01.848741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1f4d90b5-8fbe-42c5-9111-5ccaaed7040c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:01.850063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1f4d90b5-8fbe-42c5-9111-5ccaaed7040c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:01.858827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1183/2391] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:28:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:01.864054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eb0611-58a9-400c-959b-aafc3af2b960 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:01.876222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eb0611-58a9-400c-959b-aafc3af2b960 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:01.882751 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:01.883386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:01.883655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:01.883934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:01.884590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:28:01.884886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:01.886522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:01.900704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1209/2392] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:28:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:28:01.905586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4d90b5-8fbe-42c5-9111-5ccaaed7040c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:01.915340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4d90b5-8fbe-42c5-9111-5ccaaed7040c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:01.921385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:01.922049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:01.922326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:01.922598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:01.923247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:01.925448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:01.955570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1184/2393] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:28:01 2022] POST /identity/v3/projects => generated 375 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:28:01.961238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:01.973298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cab8c6b-b083-4082-a5a0-f49d2e39451b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:02.038309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:02.038928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:02.039208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:02.039483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:02.040142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:02.042236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:02.068104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1210/2394] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:28:01 2022] POST /identity/v3/users => generated 288 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:28:02.074405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:02.089416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de304049-3c8a-468c-9515-937bacd70d40 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:02.096141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:02.096774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:02.097047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:02.097344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:02.098190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:02.100026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:02.239015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1185/2395] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:28:02 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:28:02.243517 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:02.255182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc3dda1-46ae-4dd9-ad71-8e40484f2021 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:02.260216 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd89bece-d5c4-4ebe-b19e-efec532e4a5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:02.260897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd89bece-d5c4-4ebe-b19e-efec532e4a5a admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:02.261210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd89bece-d5c4-4ebe-b19e-efec532e4a5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:02.261482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd89bece-d5c4-4ebe-b19e-efec532e4a5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:02.262186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd89bece-d5c4-4ebe-b19e-efec532e4a5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:02.263863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd89bece-d5c4-4ebe-b19e-efec532e4a5a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:02.275710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1211/2396] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:28:02 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:28:02.280011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:02.292977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c8da81-3007-4e3f-aa72-f61e6267e93c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:02.297771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:02.298435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:02.298698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:02.298965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] PATH_INFO: `/v3/projects/e1d1bead2e4f4102a53f582a4815661d/users/544e6c056b854bfdb9e117be686635a0/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:02.315604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1d1bead2e4f4102a53f582a4815661d, user_id=544e6c056b854bfdb9e117be686635a0, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:02.318364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:02.333604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1186/2397] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:28:02 2022] PUT /identity/v3/projects/e1d1bead2e4f4102a53f582a4815661d/users/544e6c056b854bfdb9e117be686635a0/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:28:02.344472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8779d5-7c3a-4653-b592-46902a8a324a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:02.344754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8779d5-7c3a-4653-b592-46902a8a324a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:02.345031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8779d5-7c3a-4653-b592-46902a8a324a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:02.386658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-dd8779d5-7c3a-4653-b592-46902a8a324a None None] MFA Rules not processed for user `544e6c056b854bfdb9e117be686635a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:02.387788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dd8779d5-7c3a-4653-b592-46902a8a324a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:02.460238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1212/2398] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:28:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:02.472256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:02.474188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:02.485148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:02.540141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eebdf30-ce61-4d91-9a96-333044eff2e8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:02.544782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:02.545433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:02.545714 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:02.545972 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:02.555723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:02.557169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:02.566665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1187/2399] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:28:02 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:28:03.968422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5295977f-ee36-4e5f-939f-38f239b194cc None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:03.969024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5295977f-ee36-4e5f-939f-38f239b194cc None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:03.969024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5295977f-ee36-4e5f-939f-38f239b194cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:03.990482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5295977f-ee36-4e5f-939f-38f239b194cc None None] MFA Rules not processed for user `544e6c056b854bfdb9e117be686635a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:03.991852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5295977f-ee36-4e5f-939f-38f239b194cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:03.999772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1213/2400] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:28:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:12.319268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:12.321680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:12.413739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc572209-3db8-4dd1-9636-f41794ad766e service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:12.419973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:12.420739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:12.421106 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:12.421494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:12.423438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:12.437590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:12.439173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:12.460198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1188/2401] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:28:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:28:13.983872 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:22.454554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=57 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:33.346766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b667f3b-9391-46d2-917f-1761ffa426bd None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:33.347333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b667f3b-9391-46d2-917f-1761ffa426bd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:33.347333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b667f3b-9391-46d2-917f-1761ffa426bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:33.367877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2b667f3b-9391-46d2-917f-1761ffa426bd None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:33.369074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2b667f3b-9391-46d2-917f-1761ffa426bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:33.431842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1214/2402] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:28:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:28:33.494738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:33.494738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:33.573430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89f7b41-eb4a-4691-927e-833770aea030 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:33.580281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:33.580956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:33.581260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:33.581830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:33.594750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:33.596186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:33.605461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1189/2403] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:28:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 27 02:28:34.221072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d336e3e2-5ffd-404c-b012-5eead90c2098 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:34.221379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d336e3e2-5ffd-404c-b012-5eead90c2098 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:34.221648 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d336e3e2-5ffd-404c-b012-5eead90c2098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:34.245786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d336e3e2-5ffd-404c-b012-5eead90c2098 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:34.246987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d336e3e2-5ffd-404c-b012-5eead90c2098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:34.268082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1215/2404] 10.222.0.52 () {58 vars in 967 bytes} [Sat Aug 27 02:28:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:34.279309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:34.300104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc2f611-52d6-4134-8e0f-40b3d744d40f service swift] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:34.306699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:34.307324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:34.307594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:34.307855 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:34.316914 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:34.318837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:34.327911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1190/2405] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:28:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:28:34.578099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb096b6-6bab-4ba9-9883-6bb9c7420465 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:34.578602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb096b6-6bab-4ba9-9883-6bb9c7420465 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:34.578711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb096b6-6bab-4ba9-9883-6bb9c7420465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:34.602659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2bb096b6-6bab-4ba9-9883-6bb9c7420465 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:34.607325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2bb096b6-6bab-4ba9-9883-6bb9c7420465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:34.628975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1216/2406] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:28:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:34.645915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:34.658769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7230139b-8f45-40f6-8201-33e4cf427b25 service swift] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:34.667358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:34.667973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:34.668246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:34.668508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:34.678184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:34.679759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:34.691430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1191/2407] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:28:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:28:35.089430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f2a3f6-1aab-4441-be8e-248e6d343ef1 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:35.089718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f2a3f6-1aab-4441-be8e-248e6d343ef1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:35.089990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f2a3f6-1aab-4441-be8e-248e6d343ef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:35.113768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-59f2a3f6-1aab-4441-be8e-248e6d343ef1 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:35.114918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-59f2a3f6-1aab-4441-be8e-248e6d343ef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:35.123017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1217/2408] 10.222.0.52 () {58 vars in 967 bytes} [Sat Aug 27 02:28:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:35.136561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:35.148776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbdae2b-523b-4d68-9437-fcb57c143e44 service swift] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:35.158897 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8322fed-897d-4d44-b97f-19c969c20e0d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:35.160286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8322fed-897d-4d44-b97f-19c969c20e0d service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:35.161775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8322fed-897d-4d44-b97f-19c969c20e0d service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:35.163243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8322fed-897d-4d44-b97f-19c969c20e0d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:35.176158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8322fed-897d-4d44-b97f-19c969c20e0d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:35.177942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8322fed-897d-4d44-b97f-19c969c20e0d service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:35.186702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1192/2409] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:28:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:28:38.563186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f2a3f6-1aab-4441-be8e-248e6d343ef1 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:38.688182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f2a3f6-1aab-4441-be8e-248e6d343ef1 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:38.705770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:38.707154 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:38.708081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:38.708621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] PATH_INFO: `/v3/users/544e6c056b854bfdb9e117be686635a0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:38.711085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] RBAC: Authorizing `identity:delete_user(user_id=544e6c056b854bfdb9e117be686635a0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:38.715399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:38.775112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '544e6c056b854bfdb9e117be686635a0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:38.788335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '544e6c056b854bfdb9e117be686635a0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:38.789053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '544e6c056b854bfdb9e117be686635a0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:38.797737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '544e6c056b854bfdb9e117be686635a0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:38.805712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '544e6c056b854bfdb9e117be686635a0'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:38.811715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1218/2410] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:28:38 2022] DELETE /identity/v3/users/544e6c056b854bfdb9e117be686635a0 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:28:38.816535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:38.817082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:38.817082 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:38.841625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:38.842678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:38.917000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1193/2411] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:28:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:38.927831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:38.929680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:38.936437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:39.000773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c238c4-c580-4e01-89b2-7e66d589fa96 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:39.005568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:39.006739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:39.007044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:39.007323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:39.017944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:39.019429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:39.031939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1219/2412] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:28:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:28:39.236772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:39.238270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:39.252277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2ff7ae-332d-48b6-b7a3-79250b659686 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:39.258983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:39.259837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:39.259837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:39.259837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] PATH_INFO: `/v3/projects/e1d1bead2e4f4102a53f582a4815661d` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:39.260963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1d1bead2e4f4102a53f582a4815661d)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:39.263057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:39.313946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:39.319425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1d1bead2e4f4102a53f582a4815661d', 'request_id': 'req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:39.331388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1d1bead2e4f4102a53f582a4815661d', 'request_id': 'req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:39.339808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1d1bead2e4f4102a53f582a4815661d', 'request_id': 'req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:39.351374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] The token cache is being invalidate because project e1d1bead2e4f4102a53f582a4815661d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:28:39.351851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:39.352886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9a2ed1e6-c748-4b72-a6c5-8b04b8da4d84 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:28:39.355387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1194/2413] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:28:39 2022] DELETE /identity/v3/projects/e1d1bead2e4f4102a53f582a4815661d => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:28:43.460414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:43.463591 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:43.579046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf565b12-c4bf-483f-ba39-c4daf2eb1497 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:43.661246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:43.661895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:43.662524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:43.662524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:43.664295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:43.676982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:43.678734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:43.689554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1220/2414] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:28:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:28:49.347519 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=4,keystone:SELECT=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:53.134407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:53.135206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:53.135206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:53.156882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:28:53.158266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:53.225652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1195/2415] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:28:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:28:53.242272 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:28:53.244283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:28:53.286003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:28:53.331988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0144c76d-b45d-4bf8-9edc-a80d69cbb827 service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:28:53.338108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:28:53.338822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:28:53.338971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:28:53.339274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:28:53.348330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:28:53.350523 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:28:53.359345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1221/2416] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:28:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:29:03.219512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=25 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:03.355992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=18 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:04.873570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:04.875057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:04.961407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3991546f-9ec1-4bf1-90a0-2272939c7f21 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:04.966317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:04.967023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:04.967218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:04.967547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] PATH_INFO: `/v3/users/75744eccc3d24d4c9282cbe3a3314489` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:04.968649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=75744eccc3d24d4c9282cbe3a3314489)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:04.970725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:05.042282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75744eccc3d24d4c9282cbe3a3314489'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.051048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75744eccc3d24d4c9282cbe3a3314489'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.060341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75744eccc3d24d4c9282cbe3a3314489'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.061270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75744eccc3d24d4c9282cbe3a3314489'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.061910 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75744eccc3d24d4c9282cbe3a3314489'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.088297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1196/2417] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:29:04 2022] DELETE /identity/v3/users/75744eccc3d24d4c9282cbe3a3314489 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:29:05.353892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:05.371771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbbd82e-2daf-4f52-953c-b4ae0f9d779f service swift] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:05.461530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:05.462193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:05.462469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:05.462744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] PATH_INFO: `/v3/projects/fa43d54225bf44b283f9db5ddbbb13ca` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:05.463941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa43d54225bf44b283f9db5ddbbb13ca)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:05.466110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:05.547910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa43d54225bf44b283f9db5ddbbb13ca', 'request_id': 'req-a2f19aa1-6f25-43a2-8104-a89921e702ed'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.558124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa43d54225bf44b283f9db5ddbbb13ca', 'request_id': 'req-a2f19aa1-6f25-43a2-8104-a89921e702ed'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.564783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa43d54225bf44b283f9db5ddbbb13ca', 'request_id': 'req-a2f19aa1-6f25-43a2-8104-a89921e702ed'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.593593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] The token cache is being invalidate because project fa43d54225bf44b283f9db5ddbbb13ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:29:05.594110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.596056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a2f19aa1-6f25-43a2-8104-a89921e702ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:29:05.599613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1222/2418] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:29:05 2022] DELETE /identity/v3/projects/fa43d54225bf44b283f9db5ddbbb13ca => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:29:08.907040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:08.909076 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:08.973626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f404aca-cf8d-42e9-874e-14220e0416aa admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:09.009850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:09.033439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471e775b-e500-4c43-8923-2e5bc2fe112f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:09.034068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e775b-e500-4c43-8923-2e5bc2fe112f service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.034338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e775b-e500-4c43-8923-2e5bc2fe112f service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.034608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e775b-e500-4c43-8923-2e5bc2fe112f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.036491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-471e775b-e500-4c43-8923-2e5bc2fe112f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:09.047752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471e775b-e500-4c43-8923-2e5bc2fe112f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:09.049144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471e775b-e500-4c43-8923-2e5bc2fe112f service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:09.059370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1197/2419] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:09.228882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac03b449-39a6-4a7f-b7ae-0924a2a4ace6 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.229544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac03b449-39a6-4a7f-b7ae-0924a2a4ace6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.229544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac03b449-39a6-4a7f-b7ae-0924a2a4ace6 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.231082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1223/2420] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:29:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:29:09.238697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad233411-227c-44e8-91c6-4d023b98dcc6 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.238942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad233411-227c-44e8-91c6-4d023b98dcc6 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.239226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad233411-227c-44e8-91c6-4d023b98dcc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.265354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ad233411-227c-44e8-91c6-4d023b98dcc6 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:29:09.266611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ad233411-227c-44e8-91c6-4d023b98dcc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:09.359372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1198/2421] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:29:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:29:09.372917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac03b449-39a6-4a7f-b7ae-0924a2a4ace6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:09.375047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ac03b449-39a6-4a7f-b7ae-0924a2a4ace6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:09.440298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac03b449-39a6-4a7f-b7ae-0924a2a4ace6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:09.444186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd78c9-440a-48bc-8cbf-ea9d7d357bd4 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:09.445423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd78c9-440a-48bc-8cbf-ea9d7d357bd4 service glance] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.445752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd78c9-440a-48bc-8cbf-ea9d7d357bd4 service glance] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.446052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd78c9-440a-48bc-8cbf-ea9d7d357bd4 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.458932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd78c9-440a-48bc-8cbf-ea9d7d357bd4 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:09.460485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd78c9-440a-48bc-8cbf-ea9d7d357bd4 service glance] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:09.472530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1224/2422] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:09.549813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931eba3b-2e4e-40fc-abb8-e4f8e5617d35 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.550175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931eba3b-2e4e-40fc-abb8-e4f8e5617d35 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.550520 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931eba3b-2e4e-40fc-abb8-e4f8e5617d35 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.552262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1199/2423] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:29:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:29:09.558003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8301da4-3c4e-4722-b788-ff3810674796 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.558374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8301da4-3c4e-4722-b788-ff3810674796 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.558706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8301da4-3c4e-4722-b788-ff3810674796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.587240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c8301da4-3c4e-4722-b788-ff3810674796 None None] MFA Rules not processed for user `56aaa029a9c542e0bc4658c5cd55c2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:29:09.588456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c8301da4-3c4e-4722-b788-ff3810674796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:09.595483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1225/2424] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:29:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:29:09.619158 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931eba3b-2e4e-40fc-abb8-e4f8e5617d35 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:09.621387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-931eba3b-2e4e-40fc-abb8-e4f8e5617d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:09.636932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931eba3b-2e4e-40fc-abb8-e4f8e5617d35 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:09.640992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e43c3-9a38-47d5-b477-caf29ea4cdba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:09.641696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e43c3-9a38-47d5-b477-caf29ea4cdba service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.641984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e43c3-9a38-47d5-b477-caf29ea4cdba service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.642308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e43c3-9a38-47d5-b477-caf29ea4cdba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.652625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e43c3-9a38-47d5-b477-caf29ea4cdba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:09.655060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e43c3-9a38-47d5-b477-caf29ea4cdba service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:09.673332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1200/2425] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:09.775888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fdbcf8-a1ed-4af4-8f8c-28662f00edd3 None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.776269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fdbcf8-a1ed-4af4-8f8c-28662f00edd3 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.776555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fdbcf8-a1ed-4af4-8f8c-28662f00edd3 None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.777988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1226/2426] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:29:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:29:09.785545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8dcda4-72ea-4afc-9f45-e453f1ed16a5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.785848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8dcda4-72ea-4afc-9f45-e453f1ed16a5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.786077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8dcda4-72ea-4afc-9f45-e453f1ed16a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.813960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2f8dcda4-72ea-4afc-9f45-e453f1ed16a5 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:29:09.815250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2f8dcda4-72ea-4afc-9f45-e453f1ed16a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:09.827814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1201/2427] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:29:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:29:09.840080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fdbcf8-a1ed-4af4-8f8c-28662f00edd3 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:09.842200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f0fdbcf8-a1ed-4af4-8f8c-28662f00edd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:09.858558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fdbcf8-a1ed-4af4-8f8c-28662f00edd3 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:09.863436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:09.864067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:09.864345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:09.864708 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:09.877263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:09.878755 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:09.891791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1227/2428] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:13.041842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:13.042359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:13.042359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:13.063034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:29:13.064425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:13.136817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1202/2429] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:29:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:29:13.150227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:13.151827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:13.170836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:13.243678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6735989a-d679-4025-8f68-db91700b9fa9 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:13.249832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:13.250463 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:13.250733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:13.251043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:13.261020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:13.263150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:13.274623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1228/2430] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:29:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:29:23.128716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=75 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:23.269394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=26 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:27.414423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:27.416590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:27.433843 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b008c3-e2ce-49d4-b08b-cd25b27bec9f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:27.440532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53a3725-217b-45ec-b923-eb00722ec2c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:27.441173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53a3725-217b-45ec-b923-eb00722ec2c4 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:27.441449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53a3725-217b-45ec-b923-eb00722ec2c4 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:27.441789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53a3725-217b-45ec-b923-eb00722ec2c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:27.443665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a53a3725-217b-45ec-b923-eb00722ec2c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:27.509652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53a3725-217b-45ec-b923-eb00722ec2c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:27.511251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53a3725-217b-45ec-b923-eb00722ec2c4 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:27.522946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1203/2431] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:27 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:34.746405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:34.747943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:34.764599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35abd3-28d2-42e7-9b1c-bd2b45bf608a service swift] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:34.770691 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:34.771331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:34.771604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:34.771904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:34.774567 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:34.837067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:34.838685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:34.848025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1229/2432] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:34 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:29:37.516865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=28 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:42.406447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee427d2-f295-43ab-a72a-9dab4ca269b7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:42.407409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee427d2-f295-43ab-a72a-9dab4ca269b7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:42.408340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee427d2-f295-43ab-a72a-9dab4ca269b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:42.435787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7ee427d2-f295-43ab-a72a-9dab4ca269b7 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:29:42.437475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7ee427d2-f295-43ab-a72a-9dab4ca269b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:42.448033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1204/2433] 10.222.0.52 () {58 vars in 967 bytes} [Sat Aug 27 02:29:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:29:42.473316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:42.486760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e8ecb-94be-41db-8aff-cdfd3f96b797 service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:42.493095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:42.493859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:42.494235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:42.494614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:42.505218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:42.507195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:42.517639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1230/2434] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:29:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:29:50.235109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee427d2-f295-43ab-a72a-9dab4ca269b7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:50.249031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee427d2-f295-43ab-a72a-9dab4ca269b7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:50.254714 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:50.255309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:50.255584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:50.255878 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:50.258294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:50.326939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:50.328273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:50.338342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1205/2435] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:50 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:52.513454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=34 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:29:52.627577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:52.638087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e4d76f-c0e8-4775-afd5-3fde5e101e8b service swift] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:52.642724 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c4f3da-3ccf-4899-83a0-2e02a6238594 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:52.643319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c4f3da-3ccf-4899-83a0-2e02a6238594 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:52.643589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c4f3da-3ccf-4899-83a0-2e02a6238594 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:52.643853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c4f3da-3ccf-4899-83a0-2e02a6238594 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:52.645875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-88c4f3da-3ccf-4899-83a0-2e02a6238594 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:52.717141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c4f3da-3ccf-4899-83a0-2e02a6238594 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:52.718597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c4f3da-3ccf-4899-83a0-2e02a6238594 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:52.729102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1231/2436] 10.222.0.52 () {60 vars in 1396 bytes} [Sat Aug 27 02:29:52 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3248 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:53.959739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:53.971763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661e937f-6195-458e-93a7-be8793530cd6 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:53.978255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c71b1d-b14a-449c-b50b-b1f3ce998b99 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:53.979163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c71b1d-b14a-449c-b50b-b1f3ce998b99 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:53.979602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c71b1d-b14a-449c-b50b-b1f3ce998b99 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:53.980049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c71b1d-b14a-449c-b50b-b1f3ce998b99 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:53.982286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-63c71b1d-b14a-449c-b50b-b1f3ce998b99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:53.995971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c71b1d-b14a-449c-b50b-b1f3ce998b99 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:53.997607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c71b1d-b14a-449c-b50b-b1f3ce998b99 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:54.009920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1206/2437] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:56.650596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410cf4dc-260b-427e-ab6c-f80eab5394ca None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:56.651135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410cf4dc-260b-427e-ab6c-f80eab5394ca None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:56.651135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410cf4dc-260b-427e-ab6c-f80eab5394ca None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:56.652560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1232/2438] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:29:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:29:56.658942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78e3cce-1f15-49ac-83e6-8eacc994d660 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:56.660360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78e3cce-1f15-49ac-83e6-8eacc994d660 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:56.660360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78e3cce-1f15-49ac-83e6-8eacc994d660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:56.691963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f78e3cce-1f15-49ac-83e6-8eacc994d660 None None] MFA Rules not processed for user `56aaa029a9c542e0bc4658c5cd55c2b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:29:56.695488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f78e3cce-1f15-49ac-83e6-8eacc994d660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:56.707684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1207/2439] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:29:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:29:56.733909 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410cf4dc-260b-427e-ab6c-f80eab5394ca None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:56.747120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410cf4dc-260b-427e-ab6c-f80eab5394ca None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:56.751205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e00651-0ff7-4c46-a202-006ee855d5b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:56.751811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e00651-0ff7-4c46-a202-006ee855d5b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:56.752262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e00651-0ff7-4c46-a202-006ee855d5b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:56.752692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e00651-0ff7-4c46-a202-006ee855d5b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:56.761537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e00651-0ff7-4c46-a202-006ee855d5b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:56.763780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e00651-0ff7-4c46-a202-006ee855d5b9 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:56.774410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1233/2440] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:29:56.889443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df158a6-f51b-4cd5-9f42-0ec2ed978a14 None None] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:56.890012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df158a6-f51b-4cd5-9f42-0ec2ed978a14 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:56.890462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df158a6-f51b-4cd5-9f42-0ec2ed978a14 None None] PATH_INFO: `/` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:56.892179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1208/2441] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:29:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:29:56.898855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e8470-f721-4c55-bef1-ee1564f49715 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:56.899142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e8470-f721-4c55-bef1-ee1564f49715 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:56.899595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e8470-f721-4c55-bef1-ee1564f49715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:56.919826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-292e8470-f721-4c55-bef1-ee1564f49715 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:29:56.920976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-292e8470-f721-4c55-bef1-ee1564f49715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:29:56.935655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1234/2442] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:29:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:29:56.951789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df158a6-f51b-4cd5-9f42-0ec2ed978a14 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:29:56.967332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df158a6-f51b-4cd5-9f42-0ec2ed978a14 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:29:56.982626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d99e14b-8aaa-4c60-a1e1-76066ae4969a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:29:56.983253 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d99e14b-8aaa-4c60-a1e1-76066ae4969a service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:29:56.983524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d99e14b-8aaa-4c60-a1e1-76066ae4969a service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:29:56.983787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d99e14b-8aaa-4c60-a1e1-76066ae4969a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:29:56.995226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d99e14b-8aaa-4c60-a1e1-76066ae4969a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:29:56.996885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d99e14b-8aaa-4c60-a1e1-76066ae4969a service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:29:57.009478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1209/2443] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:29:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:30:04.619617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:30:04.620225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:30:04.620330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:30:04.640995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] MFA Rules not processed for user `d945d512ae034ec08838d0a3afde2429`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:30:04.641504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:30:04.650763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1235/2444] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:30:04 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:30:07.003485 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=46 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:30:08.886051 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aefeab1-7efc-446f-8f88-7788f569cfa0 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:30:08.886739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aefeab1-7efc-446f-8f88-7788f569cfa0 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:30:08.886739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aefeab1-7efc-446f-8f88-7788f569cfa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:30:08.906586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5aefeab1-7efc-446f-8f88-7788f569cfa0 None None] MFA Rules not processed for user `26a86351583d49d090b3db09f127d49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:30:08.907854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5aefeab1-7efc-446f-8f88-7788f569cfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:30:08.915819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1210/2445] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:30:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:30:14.634414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=40 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:30:18.901758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:06.101428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:06.102939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:06.280546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc94fb39-81f7-45ce-81b1-4e3bcf3288d3 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:06.286196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:06.286996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:06.287265 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:06.287531 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:06.289400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:06.306318 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:06.308879 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:06.319308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1236/2446] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:31:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:31:10.691014 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:10.691614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:10.691614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:10.710698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] MFA Rules not processed for user `d750e6e172d34d4a9333e0c059c5442b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:10.711893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:10.719497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1211/2447] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:10 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:31:16.315259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=63 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:20.705918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:42.840687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:42.861651 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8001694-decc-49c1-bf10-05a915d9b14f service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:42.865797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:42.866409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:42.866687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:42.866962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:42.868874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:42.970088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:42.971479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:42.993663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1237/2448] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:31:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:31:46.320202 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:46.321966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:46.338658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58398558-14c4-4f6e-ba50-00ec78ac2fff None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:46.344949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:46.345597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:46.345884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:46.346120 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:46.347980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:46.360719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:46.362529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:46.375730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1212/2449] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:31:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:31:46.382820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:46.384331 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:46.411680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99589b72-479f-44cd-8f43-c3f619a4b796 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:46.417302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:46.418657 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:46.419124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:46.419402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] PATH_INFO: `/v3/users/d945d512ae034ec08838d0a3afde2429` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:46.420605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d945d512ae034ec08838d0a3afde2429)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:46.422837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:46.479468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd945d512ae034ec08838d0a3afde2429'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.494666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd945d512ae034ec08838d0a3afde2429'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.496074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd945d512ae034ec08838d0a3afde2429'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.504196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd945d512ae034ec08838d0a3afde2429'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.512703 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e9798311-02e8-441a-a632-ac7e725eb807 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd945d512ae034ec08838d0a3afde2429'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.519858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1238/2450] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:31:46 2022] DELETE /identity/v3/users/d945d512ae034ec08838d0a3afde2429 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:31:46.706673 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:46.721228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027da237-a215-43ef-b777-fe95a530f50e service placement] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:46.785180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:46.785987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:46.786210 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:46.786462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] PATH_INFO: `/v3/projects/880ac4003248441c809a76cfd1b221eb` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:46.787860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=880ac4003248441c809a76cfd1b221eb)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:46.790023 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:46.855590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '880ac4003248441c809a76cfd1b221eb', 'request_id': 'req-8a3a98ac-248a-40fa-a67a-e8f45747aff2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.864575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '880ac4003248441c809a76cfd1b221eb', 'request_id': 'req-8a3a98ac-248a-40fa-a67a-e8f45747aff2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.871992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '880ac4003248441c809a76cfd1b221eb', 'request_id': 'req-8a3a98ac-248a-40fa-a67a-e8f45747aff2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.882909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] The token cache is being invalidate because project 880ac4003248441c809a76cfd1b221eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:31:46.883188 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.883840 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8a3a98ac-248a-40fa-a67a-e8f45747aff2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:46.885404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1213/2451] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:31:46 2022] DELETE /identity/v3/projects/880ac4003248441c809a76cfd1b221eb => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:31:46.890830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176dc34-c7ea-4b82-b24f-85632152252d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:46.891132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176dc34-c7ea-4b82-b24f-85632152252d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:46.891401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176dc34-c7ea-4b82-b24f-85632152252d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:46.939071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:46.964039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b176dc34-c7ea-4b82-b24f-85632152252d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:46.966326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b176dc34-c7ea-4b82-b24f-85632152252d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:47.035078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1239/2452] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:31:47.044745 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ecdb57-6e56-4498-a428-900d6c40cb2a None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.044949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ecdb57-6e56-4498-a428-900d6c40cb2a None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.045248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ecdb57-6e56-4498-a428-900d6c40cb2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.065289 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-65ecdb57-6e56-4498-a428-900d6c40cb2a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:47.066574 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-65ecdb57-6e56-4498-a428-900d6c40cb2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:47.075712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1214/2453] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:31:47.080136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b176dc34-c7ea-4b82-b24f-85632152252d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:47.093793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b176dc34-c7ea-4b82-b24f-85632152252d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:47.099279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:47.099882 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.100157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.100426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.101075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:31:47.101419 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:47.103016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:47.116363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1240/2454] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:31:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:31:47.121313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ecdb57-6e56-4498-a428-900d6c40cb2a None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:47.131372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ecdb57-6e56-4498-a428-900d6c40cb2a None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:47.137518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:47.138184 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.138440 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.138701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.139339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:47.141408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:47.172678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1215/2455] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:31:47 2022] POST /identity/v3/projects => generated 371 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:31:47.177509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:47.189674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9149754e-256f-42e3-b9bc-d04b99fe8509 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:47.256148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:47.256749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.257026 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.257326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.257978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:47.259891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:47.282654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1241/2456] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:31:47 2022] POST /identity/v3/users => generated 286 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:31:47.286832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:47.299154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4575babd-e563-4927-8cd5-8e2ff88ab5be admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:47.305747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:47.306556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.307065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.307452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.308306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:47.310187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:47.321189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1216/2457] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:31:47 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:31:47.325705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:47.336771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc1b948-8d82-4da7-9e41-b5734d0f4c70 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:47.342849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ff7e2d-9eb8-4ed8-89ea-652cf7780fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:47.343457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ff7e2d-9eb8-4ed8-89ea-652cf7780fe0 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.343727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ff7e2d-9eb8-4ed8-89ea-652cf7780fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.343990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ff7e2d-9eb8-4ed8-89ea-652cf7780fe0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.344635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ff7e2d-9eb8-4ed8-89ea-652cf7780fe0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:47.346336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ff7e2d-9eb8-4ed8-89ea-652cf7780fe0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:47.359481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1242/2458] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:31:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:31:47.366653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:47.378795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b4d6c-d01f-4070-bfa4-7157a6840182 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:47.384524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:47.385209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.385489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.385766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] PATH_INFO: `/v3/projects/21f532fa34834346bcceca9a40a7fc09/users/ccdb258153c84117aa02b27aa8e79a79/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.402009 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21f532fa34834346bcceca9a40a7fc09, user_id=ccdb258153c84117aa02b27aa8e79a79, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:47.404639 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:47.420302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1217/2459] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:31:47 2022] PUT /identity/v3/projects/21f532fa34834346bcceca9a40a7fc09/users/ccdb258153c84117aa02b27aa8e79a79/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:31:47.431316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bbb4d2-7cbf-4030-84f5-a3de3395440b None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.431857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bbb4d2-7cbf-4030-84f5-a3de3395440b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.432000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bbb4d2-7cbf-4030-84f5-a3de3395440b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.465716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-74bbb4d2-7cbf-4030-84f5-a3de3395440b None None] MFA Rules not processed for user `ccdb258153c84117aa02b27aa8e79a79`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:47.467290 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-74bbb4d2-7cbf-4030-84f5-a3de3395440b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:47.524030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:47.536524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1243/2460] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:47 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:31:47.549263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:47.551577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:47.574161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:47.670441 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ec720-b0a4-411f-aaf4-4ed4327fc631 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:47.677154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c1961f-8ed9-41e3-9310-b7c309440757 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:47.677774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1961f-8ed9-41e3-9310-b7c309440757 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:47.678049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1961f-8ed9-41e3-9310-b7c309440757 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:47.678315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1961f-8ed9-41e3-9310-b7c309440757 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:47.686871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c1961f-8ed9-41e3-9310-b7c309440757 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:47.688447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c1961f-8ed9-41e3-9310-b7c309440757 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:47.697582 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1218/2461] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:31:47 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:31:50.434199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bbb4d2-7cbf-4030-84f5-a3de3395440b None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:50.503187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bbb4d2-7cbf-4030-84f5-a3de3395440b None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:50.508563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:50.509233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:50.509559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:50.509801 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] PATH_INFO: `/v3/users/ccdb258153c84117aa02b27aa8e79a79` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:50.510991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccdb258153c84117aa02b27aa8e79a79)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:50.513805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:50.563978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccdb258153c84117aa02b27aa8e79a79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:50.576539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccdb258153c84117aa02b27aa8e79a79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:50.577256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccdb258153c84117aa02b27aa8e79a79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:50.586170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccdb258153c84117aa02b27aa8e79a79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:50.593562 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccdb258153c84117aa02b27aa8e79a79'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:50.599718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1244/2462] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:31:50 2022] DELETE /identity/v3/users/ccdb258153c84117aa02b27aa8e79a79 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:31:50.604550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:50.605249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:50.605249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:50.626881 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:50.628134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:50.696671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1219/2463] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:31:50.707383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:50.708919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:50.786563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b912c4-fbaf-4c7b-bd6a-0a94d1f3ed3f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:50.790794 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9257eeb9-ae1d-47a2-9d70-47d35ba71e49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:50.791407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257eeb9-ae1d-47a2-9d70-47d35ba71e49 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:50.791688 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257eeb9-ae1d-47a2-9d70-47d35ba71e49 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:50.791940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257eeb9-ae1d-47a2-9d70-47d35ba71e49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:50.806624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9257eeb9-ae1d-47a2-9d70-47d35ba71e49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:50.808324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9257eeb9-ae1d-47a2-9d70-47d35ba71e49 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:50.819244 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1245/2464] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:31:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:31:51.120944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.122660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:51.135816 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b180c397-a8b3-4c0e-907a-49fff11235bd None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.141430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.142118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.142490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.142818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] PATH_INFO: `/v3/projects/21f532fa34834346bcceca9a40a7fc09` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.144277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] RBAC: Authorizing `identity:delete_project(project_id=21f532fa34834346bcceca9a40a7fc09)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.146296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.203911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21f532fa34834346bcceca9a40a7fc09', 'request_id': 'req-babf2fe8-cd0f-444d-906c-04281f36080c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:51.212347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21f532fa34834346bcceca9a40a7fc09', 'request_id': 'req-babf2fe8-cd0f-444d-906c-04281f36080c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:51.219720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21f532fa34834346bcceca9a40a7fc09', 'request_id': 'req-babf2fe8-cd0f-444d-906c-04281f36080c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:51.224591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:51.232295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] The token cache is being invalidate because project 21f532fa34834346bcceca9a40a7fc09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:31:51.232845 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:51.233890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-babf2fe8-cd0f-444d-906c-04281f36080c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:31:51.235632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1220/2465] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:31:51 2022] DELETE /identity/v3/projects/21f532fa34834346bcceca9a40a7fc09 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:31:51.241633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb1cb86-0048-441b-ba91-1207202a98d9 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.241913 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb1cb86-0048-441b-ba91-1207202a98d9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.242178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb1cb86-0048-441b-ba91-1207202a98d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.251646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:51.265704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4eb1cb86-0048-441b-ba91-1207202a98d9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:51.266926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4eb1cb86-0048-441b-ba91-1207202a98d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:51.328926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1246/2466] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:31:51.339212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272386cf-4dc6-4a3e-98bf-496be4cd2eed None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.339491 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272386cf-4dc6-4a3e-98bf-496be4cd2eed None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.339789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272386cf-4dc6-4a3e-98bf-496be4cd2eed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.363517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-272386cf-4dc6-4a3e-98bf-496be4cd2eed None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:51.364746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-272386cf-4dc6-4a3e-98bf-496be4cd2eed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:51.373917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1221/2467] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:31:51.379180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb1cb86-0048-441b-ba91-1207202a98d9 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.391605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb1cb86-0048-441b-ba91-1207202a98d9 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.396846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.397490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.397762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.398024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.398695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:31:51.398991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.400586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.413073 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1247/2468] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:31:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:31:51.418047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272386cf-4dc6-4a3e-98bf-496be4cd2eed None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.427631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272386cf-4dc6-4a3e-98bf-496be4cd2eed None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.432540 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.433239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.433669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.434222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.435062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.436986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.473093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1222/2469] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:31:51 2022] POST /identity/v3/projects => generated 367 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:31:51.477878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.488947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185fa180-e13a-449e-8a99-d0450acb1d76 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.550959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.551647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.552188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.552188 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.552827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.554785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.580569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1248/2470] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:31:51 2022] POST /identity/v3/users => generated 284 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:31:51.581835 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.593021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76d2274-60a0-4dcd-9da8-99e4b347f164 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.599870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.600419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.600622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.600933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.601864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.604264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.616126 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1223/2471] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:31:51 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:31:51.620597 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.632086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b779531-6878-4538-bd52-d1365f62c4dc admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.636878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948a54f0-4d11-4355-b53e-7c58e06eed83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.637521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a54f0-4d11-4355-b53e-7c58e06eed83 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.637795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a54f0-4d11-4355-b53e-7c58e06eed83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.638062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a54f0-4d11-4355-b53e-7c58e06eed83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.638710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948a54f0-4d11-4355-b53e-7c58e06eed83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.640351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948a54f0-4d11-4355-b53e-7c58e06eed83 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.651960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1249/2472] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:31:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:31:51.656633 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.671671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a5602a-773e-4dac-a54d-9e11d23e8527 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.677044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.677850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.678292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.678577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] PATH_INFO: `/v3/projects/3edf545a1244492bbe69bde2d1997211/users/8f2a5c701bdd433fa9441cc3b462429f/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.694365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3edf545a1244492bbe69bde2d1997211, user_id=8f2a5c701bdd433fa9441cc3b462429f, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.697010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.712031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1224/2473] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:31:51 2022] PUT /identity/v3/projects/3edf545a1244492bbe69bde2d1997211/users/8f2a5c701bdd433fa9441cc3b462429f/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:31:51.724292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1be72d3-4fd7-4ca2-bf2f-f38867ccd9b5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.724622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1be72d3-4fd7-4ca2-bf2f-f38867ccd9b5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.724902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1be72d3-4fd7-4ca2-bf2f-f38867ccd9b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.756574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d1be72d3-4fd7-4ca2-bf2f-f38867ccd9b5 None None] MFA Rules not processed for user `8f2a5c701bdd433fa9441cc3b462429f`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:31:51.758065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d1be72d3-4fd7-4ca2-bf2f-f38867ccd9b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:51.824941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:31:51.829341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1250/2474] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:31:51 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:31:51.842687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:31:51.845625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:31:51.934364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17718f3a-a5aa-466c-9cdd-86467260ca5b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:31:51.938607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bd8cdb-7dda-4325-8350-1652ab48182a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:31:51.939220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd8cdb-7dda-4325-8350-1652ab48182a service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:31:51.939494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd8cdb-7dda-4325-8350-1652ab48182a service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:31:51.939763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd8cdb-7dda-4325-8350-1652ab48182a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:31:51.949280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bd8cdb-7dda-4325-8350-1652ab48182a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:31:51.950716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bd8cdb-7dda-4325-8350-1652ab48182a service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:31:51.959999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1225/2475] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:31:51 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:00.674357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1be72d3-4fd7-4ca2-bf2f-f38867ccd9b5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:00.777416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1be72d3-4fd7-4ca2-bf2f-f38867ccd9b5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:00.783361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:00.784270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:00.784761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:00.785289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] PATH_INFO: `/v3/users/8f2a5c701bdd433fa9441cc3b462429f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:00.786886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f2a5c701bdd433fa9441cc3b462429f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:00.789066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:00.854199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f2a5c701bdd433fa9441cc3b462429f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:00.873876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f2a5c701bdd433fa9441cc3b462429f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:00.874923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f2a5c701bdd433fa9441cc3b462429f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:00.884629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f2a5c701bdd433fa9441cc3b462429f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:00.894477 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f2a5c701bdd433fa9441cc3b462429f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:00.902725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1251/2476] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:00 2022] DELETE /identity/v3/users/8f2a5c701bdd433fa9441cc3b462429f => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:00.907337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:00.907872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:00.907872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:00.934920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:00.936247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:01.017101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1226/2477] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:01.032468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:01.034194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:01.108955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8c0196-b2b0-48f6-b538-91e8282c6f25 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:01.113197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cef664-62ae-44b3-8555-53119b9d43fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:01.113928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cef664-62ae-44b3-8555-53119b9d43fe service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.114296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cef664-62ae-44b3-8555-53119b9d43fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.114682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cef664-62ae-44b3-8555-53119b9d43fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.125527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cef664-62ae-44b3-8555-53119b9d43fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:01.127177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cef664-62ae-44b3-8555-53119b9d43fe service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:01.141745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1252/2478] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:32:01.455776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:01.457642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:01.471467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad22c014-f1d0-4617-a733-531fd1d011e2 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:01.476490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:01.477147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.477425 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.477689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] PATH_INFO: `/v3/projects/3edf545a1244492bbe69bde2d1997211` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.478826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3edf545a1244492bbe69bde2d1997211)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:01.480744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:01.492359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:01.534231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3edf545a1244492bbe69bde2d1997211', 'request_id': 'req-67e79ffb-329b-4a1b-91e3-a425303a8890'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:01.543499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3edf545a1244492bbe69bde2d1997211', 'request_id': 'req-67e79ffb-329b-4a1b-91e3-a425303a8890'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:01.550615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3edf545a1244492bbe69bde2d1997211', 'request_id': 'req-67e79ffb-329b-4a1b-91e3-a425303a8890'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:01.561623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] The token cache is being invalidate because project 3edf545a1244492bbe69bde2d1997211 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:01.561911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:01.562592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-67e79ffb-329b-4a1b-91e3-a425303a8890 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:01.564089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1227/2479] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:01 2022] DELETE /identity/v3/projects/3edf545a1244492bbe69bde2d1997211 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:32:01.569069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912abf92-3602-42bd-aac0-dfb86c86cd37 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.569457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912abf92-3602-42bd-aac0-dfb86c86cd37 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.569775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912abf92-3602-42bd-aac0-dfb86c86cd37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.590552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-912abf92-3602-42bd-aac0-dfb86c86cd37 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:01.592099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-912abf92-3602-42bd-aac0-dfb86c86cd37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:01.611780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:01.659296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1253/2480] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:01.667171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24429fb9-a600-46ac-bc14-87eeea31b5da None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.667453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24429fb9-a600-46ac-bc14-87eeea31b5da None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.667721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24429fb9-a600-46ac-bc14-87eeea31b5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.689358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-24429fb9-a600-46ac-bc14-87eeea31b5da None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:01.690550 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-24429fb9-a600-46ac-bc14-87eeea31b5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:01.698948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1228/2481] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:01.703485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912abf92-3602-42bd-aac0-dfb86c86cd37 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:01.714510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912abf92-3602-42bd-aac0-dfb86c86cd37 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:01.719799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:01.720493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.720836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.721107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.721799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:32:01.722100 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:01.723694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:01.736386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1254/2482] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:32:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 27 02:32:01.740709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24429fb9-a600-46ac-bc14-87eeea31b5da None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:01.750778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24429fb9-a600-46ac-bc14-87eeea31b5da None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:01.759911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:01.760985 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.761512 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.762405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.763100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:01.766237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:01.805071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1229/2483] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:01 2022] POST /identity/v3/projects => generated 373 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:01.809813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:01.821025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ffc2ac-5ebe-4f18-be7f-ea75c52a21fb admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:01.882271 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:01.882898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.883180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.883457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.884117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:01.886835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:01.914784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1255/2484] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:01 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:01.916602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:01.927824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66624cf2-6293-46ff-ba24-82891502803e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:01.933226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:01.934211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.934565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.934849 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.935488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:01.937077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:01.950298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1230/2485] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:32:01.954707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:01.965392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd0152-9330-4da3-b26a-3842221f2e71 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:01.970668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e937790c-e60e-478b-b09a-d7a4584bfa5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:01.971267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e937790c-e60e-478b-b09a-d7a4584bfa5a admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:01.971550 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e937790c-e60e-478b-b09a-d7a4584bfa5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:01.971802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e937790c-e60e-478b-b09a-d7a4584bfa5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:01.972447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e937790c-e60e-478b-b09a-d7a4584bfa5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:01.974078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e937790c-e60e-478b-b09a-d7a4584bfa5a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:01.984825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1256/2486] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:01 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:01.989842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:02.004743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fb874b-fa78-4b9f-90e7-d4906e2d8807 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:02.013074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:02.013819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:02.014088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:02.014355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] PATH_INFO: `/v3/projects/2ce29c5c49b94cb2bb257638c12184e3/users/dab0e4d69801443fab295701430218d8/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:02.033385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ce29c5c49b94cb2bb257638c12184e3, user_id=dab0e4d69801443fab295701430218d8, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:02.036032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:02.051729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1231/2487] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:01 2022] PUT /identity/v3/projects/2ce29c5c49b94cb2bb257638c12184e3/users/dab0e4d69801443fab295701430218d8/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:02.069182 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2306c6-792e-4f23-860e-d1b6f5c4b870 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:02.069468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2306c6-792e-4f23-860e-d1b6f5c4b870 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:02.069739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2306c6-792e-4f23-860e-d1b6f5c4b870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:02.102791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3d2306c6-792e-4f23-860e-d1b6f5c4b870 None None] MFA Rules not processed for user `dab0e4d69801443fab295701430218d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:02.104070 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3d2306c6-792e-4f23-860e-d1b6f5c4b870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:02.185299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1257/2488] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:02.197776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:02.200122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:02.262543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1721dbf-95b1-4c7f-b9e9-12653edcfe37 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:02.266698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8234821-c1ab-4799-be5e-6764459cc123 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:02.267269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8234821-c1ab-4799-be5e-6764459cc123 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:02.267537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8234821-c1ab-4799-be5e-6764459cc123 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:02.267813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8234821-c1ab-4799-be5e-6764459cc123 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:02.281570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8234821-c1ab-4799-be5e-6764459cc123 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:02.283037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8234821-c1ab-4799-be5e-6764459cc123 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:02.292319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1232/2489] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:02 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:02.314992 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2306c6-792e-4f23-860e-d1b6f5c4b870 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:02.334865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:02.379313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2306c6-792e-4f23-860e-d1b6f5c4b870 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:02.384274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:02.384890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:02.385174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:02.385447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] PATH_INFO: `/v3/users/dab0e4d69801443fab295701430218d8` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:02.386658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=dab0e4d69801443fab295701430218d8)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:02.388752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:02.437762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dab0e4d69801443fab295701430218d8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:02.449234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dab0e4d69801443fab295701430218d8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:02.449975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dab0e4d69801443fab295701430218d8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:02.458907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dab0e4d69801443fab295701430218d8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:02.468593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dab0e4d69801443fab295701430218d8'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:02.476999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1258/2490] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:02 2022] DELETE /identity/v3/users/dab0e4d69801443fab295701430218d8 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:02.482677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:02.482949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:02.483216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:02.505709 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:02.507118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:02.550069 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:02.579112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1233/2491] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:02.589823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:02.591600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:02.657956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b671c88-bd59-4526-a006-89e995d4ce3e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:02.662335 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d142715-38c1-4d02-b260-7664b5ac5c9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:02.662952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d142715-38c1-4d02-b260-7664b5ac5c9b service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:02.663227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d142715-38c1-4d02-b260-7664b5ac5c9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:02.663497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d142715-38c1-4d02-b260-7664b5ac5c9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:02.674086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d142715-38c1-4d02-b260-7664b5ac5c9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:02.675521 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d142715-38c1-4d02-b260-7664b5ac5c9b service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:02.685264 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1259/2492] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:02.980498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:02.982181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:02.998064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d21e29-ea02-44d3-9868-9a2de6273d53 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.003694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.004398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.004670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.004935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] PATH_INFO: `/v3/projects/2ce29c5c49b94cb2bb257638c12184e3` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.006182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ce29c5c49b94cb2bb257638c12184e3)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.008140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.064442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ce29c5c49b94cb2bb257638c12184e3', 'request_id': 'req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:03.072717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ce29c5c49b94cb2bb257638c12184e3', 'request_id': 'req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:03.079706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ce29c5c49b94cb2bb257638c12184e3', 'request_id': 'req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:03.090936 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] The token cache is being invalidate because project 2ce29c5c49b94cb2bb257638c12184e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:03.091304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:03.092330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-13a93ed0-59d5-47f0-9b7e-bcd2574a286d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:03.093957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1234/2493] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:02 2022] DELETE /identity/v3/projects/2ce29c5c49b94cb2bb257638c12184e3 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:03.098914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae043019-d645-4567-a404-5dd49038b117 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.099407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae043019-d645-4567-a404-5dd49038b117 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.099516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae043019-d645-4567-a404-5dd49038b117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.118895 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ae043019-d645-4567-a404-5dd49038b117 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:03.120099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ae043019-d645-4567-a404-5dd49038b117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:03.162901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:03.209806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1260/2494] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:32:03.219661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a63882f-17d5-4db5-8a02-f3f8daf937ce None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.219939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a63882f-17d5-4db5-8a02-f3f8daf937ce None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.220230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a63882f-17d5-4db5-8a02-f3f8daf937ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.240966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7a63882f-17d5-4db5-8a02-f3f8daf937ce None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:03.242191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7a63882f-17d5-4db5-8a02-f3f8daf937ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:03.250743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1235/2495] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:32:03.256716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae043019-d645-4567-a404-5dd49038b117 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:03.266869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae043019-d645-4567-a404-5dd49038b117 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.271436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.272028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.272299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.272557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.273245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:32:03.273624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.275195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.286939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1261/2496] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:32:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:32:03.291385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a63882f-17d5-4db5-8a02-f3f8daf937ce None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:03.302857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a63882f-17d5-4db5-8a02-f3f8daf937ce None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.311189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.312058 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.312447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.312828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.313851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.316786 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.352247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1236/2497] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:03 2022] POST /identity/v3/projects => generated 377 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:03.353460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:03.363811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11926f22-7035-46c7-9934-1f2db382ee2e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.421518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.422141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.422416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.422682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.423339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.425359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.450529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1262/2498] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:03 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:03.454739 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:03.465913 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5d2426-9d71-4dda-85df-ce8767fe61ee admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.472319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.473181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.473660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.473889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.474530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.476144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.487933 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1237/2499] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:32:03.492342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:03.506638 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85769c89-8eff-4c2d-837d-96985d356587 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.511641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7261b2-9ad6-4448-906a-e2bc448efdbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.512804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7261b2-9ad6-4448-906a-e2bc448efdbb admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.512804 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7261b2-9ad6-4448-906a-e2bc448efdbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.515685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7261b2-9ad6-4448-906a-e2bc448efdbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.515685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7261b2-9ad6-4448-906a-e2bc448efdbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.516381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7261b2-9ad6-4448-906a-e2bc448efdbb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.528961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1263/2500] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:03 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:32:03.534231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:03.560884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5eabdc-c2cf-448c-98db-ec9946ac8295 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.564915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.565736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.566009 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.566432 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] PATH_INFO: `/v3/projects/d938e2c2df374596be306bb133ec8322/users/04c4fdaf3d664091aa6f344f73ce6b19/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.585949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d938e2c2df374596be306bb133ec8322, user_id=04c4fdaf3d664091aa6f344f73ce6b19, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.588943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.606068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1238/2501] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:03 2022] PUT /identity/v3/projects/d938e2c2df374596be306bb133ec8322/users/04c4fdaf3d664091aa6f344f73ce6b19/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:32:03.616683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.617064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.617626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.650700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] MFA Rules not processed for user `04c4fdaf3d664091aa6f344f73ce6b19`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:03.652261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:03.722456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1264/2502] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:03.734500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:03.736320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:03.797148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f101236-e3c5-4a94-9d0a-e97ad9427a93 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:03.801389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:03.802011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:03.802280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:03.802543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:03.811711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:03.813171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:03.817870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:03.827038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1239/2503] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:03 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:05.990647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:05.992074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:06.034789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:06.057326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0aa38e-c375-4782-b497-437df28e2e83 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:06.061584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:06.062213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:06.062488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:06.062754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:06.064797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:06.136342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:06.137881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:06.148524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1265/2504] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:09.897765 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:09.899005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:09.983975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0874ed6-c834-4e0d-9f78-ccda740b34bd service glance] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:09.991581 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:09.992231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:09.992531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:09.992794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:09.994661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:10.007397 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:10.008974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:10.023031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1240/2505] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:14.184280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:14.187184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:14.205456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de372b4-00eb-4c69-ba26-81b8fa0b582a service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:14.210353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:14.211009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:14.211284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:14.211557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] PATH_INFO: `/v3/users/642291c4daa04a6ab3fbce92a8be39c6` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:14.212660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] RBAC: Authorizing `identity:delete_user(user_id=642291c4daa04a6ab3fbce92a8be39c6)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:14.214621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:14.266324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '642291c4daa04a6ab3fbce92a8be39c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.279203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '642291c4daa04a6ab3fbce92a8be39c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.279925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '642291c4daa04a6ab3fbce92a8be39c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.288869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '642291c4daa04a6ab3fbce92a8be39c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.297774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '642291c4daa04a6ab3fbce92a8be39c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.304406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1266/2506] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:14 2022] DELETE /identity/v3/users/642291c4daa04a6ab3fbce92a8be39c6 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:14.508832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:14.526224 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd6adf-122f-48e8-ac78-6a4ea713950a service placement] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:14.595451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:14.596500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:14.596500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:14.596682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] PATH_INFO: `/v3/projects/b2ddff853b7546788b645a0344c2297b` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:14.597773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b2ddff853b7546788b645a0344c2297b)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:14.599676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:14.657271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b2ddff853b7546788b645a0344c2297b', 'request_id': 'req-8495e156-6f7d-46cc-928a-8d13d5beef2e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.667865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b2ddff853b7546788b645a0344c2297b', 'request_id': 'req-8495e156-6f7d-46cc-928a-8d13d5beef2e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.675379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b2ddff853b7546788b645a0344c2297b', 'request_id': 'req-8495e156-6f7d-46cc-928a-8d13d5beef2e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.686553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] The token cache is being invalidate because project b2ddff853b7546788b645a0344c2297b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:14.686842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.687488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8495e156-6f7d-46cc-928a-8d13d5beef2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.688978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1241/2507] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:14 2022] DELETE /identity/v3/projects/b2ddff853b7546788b645a0344c2297b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:14.692964 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:14.694468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:14.707749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46517454-ae3f-4ec0-b48d-b084679f8d3f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:14.761056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:14.768327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:14.768970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:14.769485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:14.770145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] PATH_INFO: `/v3/users/399110b53734477c98318822e79eb390` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:14.771488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] RBAC: Authorizing `identity:delete_user(user_id=399110b53734477c98318822e79eb390)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:14.773601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:14.822604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '399110b53734477c98318822e79eb390'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.834874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '399110b53734477c98318822e79eb390'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.835662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '399110b53734477c98318822e79eb390'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.843802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '399110b53734477c98318822e79eb390'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.850644 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '399110b53734477c98318822e79eb390'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:14.856491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1267/2508] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:14 2022] DELETE /identity/v3/users/399110b53734477c98318822e79eb390 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:15.041220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:15.042970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:15.044098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:15.044282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:15.044802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:15.066018 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] MFA Rules not processed for user `26a86351583d49d090b3db09f127d49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:15.067346 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:15.081421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:15.114112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a6ed18-8517-4fd3-b679-783044a97771 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:15.122555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:15.123325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:15.123617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:15.123899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] PATH_INFO: `/v3/projects/22afc26b2f4243289fd730ec223b2cca` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:15.125350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] RBAC: Authorizing `identity:delete_project(project_id=22afc26b2f4243289fd730ec223b2cca)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:15.127356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:15.137284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1242/2509] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:15 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:15.181607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '22afc26b2f4243289fd730ec223b2cca', 'request_id': 'req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:15.189929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '22afc26b2f4243289fd730ec223b2cca', 'request_id': 'req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:15.196052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '22afc26b2f4243289fd730ec223b2cca', 'request_id': 'req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:15.209174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] The token cache is being invalidate because project 22afc26b2f4243289fd730ec223b2cca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:15.209464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:15.210080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-5f90f5aa-f4d4-4ec8-9c6f-8c608912678b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:15.211555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1268/2510] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:15 2022] DELETE /identity/v3/projects/22afc26b2f4243289fd730ec223b2cca => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:25.126541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=16 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:25.206051 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=78,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:29.161149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:29.162670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:29.177388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82d4ea5-34f6-470e-a130-8a63fafd85bf None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:29.251198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:29.251730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:29.252021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:29.252261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] PATH_INFO: `/v3/users/04c4fdaf3d664091aa6f344f73ce6b19` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:29.253577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04c4fdaf3d664091aa6f344f73ce6b19)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:29.255753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:29.314357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04c4fdaf3d664091aa6f344f73ce6b19'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.322373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04c4fdaf3d664091aa6f344f73ce6b19'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.330610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04c4fdaf3d664091aa6f344f73ce6b19'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.331272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04c4fdaf3d664091aa6f344f73ce6b19'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.331940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04c4fdaf3d664091aa6f344f73ce6b19'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.351882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1243/2511] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:29 2022] DELETE /identity/v3/users/04c4fdaf3d664091aa6f344f73ce6b19 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:29.357906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:29.358437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:29.358542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:29.382203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:29.383399 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:29.444103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1269/2512] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:29.455167 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:29.456687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:29.517326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f162596e-28f6-4a38-aacb-44606b31f572 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:29.521319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d85591e-e3ca-484a-8eab-9d2d0d4de472 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:29.521946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85591e-e3ca-484a-8eab-9d2d0d4de472 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:29.522209 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85591e-e3ca-484a-8eab-9d2d0d4de472 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:29.522479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85591e-e3ca-484a-8eab-9d2d0d4de472 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:29.532660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d85591e-e3ca-484a-8eab-9d2d0d4de472 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:29.534149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d85591e-e3ca-484a-8eab-9d2d0d4de472 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:29.544783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1244/2513] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:29.822461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:29.823931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:29.836907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcefa4e2-8010-4be3-aa65-b3252d3bb6bd None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:29.844234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:29.844884 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:29.845155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:29.845446 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] PATH_INFO: `/v3/projects/d938e2c2df374596be306bb133ec8322` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:29.846624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d938e2c2df374596be306bb133ec8322)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:29.848546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:29.897088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd938e2c2df374596be306bb133ec8322', 'request_id': 'req-6610c2cb-3124-44fd-be14-d875b42dfb20'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.903792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd938e2c2df374596be306bb133ec8322', 'request_id': 'req-6610c2cb-3124-44fd-be14-d875b42dfb20'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.910656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd938e2c2df374596be306bb133ec8322', 'request_id': 'req-6610c2cb-3124-44fd-be14-d875b42dfb20'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.925268 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] The token cache is being invalidate because project d938e2c2df374596be306bb133ec8322 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:29.925557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.926241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6610c2cb-3124-44fd-be14-d875b42dfb20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:29.927774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1270/2514] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:29 2022] DELETE /identity/v3/projects/d938e2c2df374596be306bb133ec8322 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:29.932774 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53678236-a84e-4ce6-8736-31b77bb727e5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:29.933053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53678236-a84e-4ce6-8736-31b77bb727e5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:29.933356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53678236-a84e-4ce6-8736-31b77bb727e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:29.954363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-53678236-a84e-4ce6-8736-31b77bb727e5 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:29.955648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-53678236-a84e-4ce6-8736-31b77bb727e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:29.975193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:30.021011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1245/2515] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:30.030526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d35829b-f298-4521-98e6-c9a67beb08a9 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.030809 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d35829b-f298-4521-98e6-c9a67beb08a9 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.031079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d35829b-f298-4521-98e6-c9a67beb08a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.051410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2d35829b-f298-4521-98e6-c9a67beb08a9 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:30.052514 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2d35829b-f298-4521-98e6-c9a67beb08a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:30.061030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1271/2516] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:30.065475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53678236-a84e-4ce6-8736-31b77bb727e5 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.076813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53678236-a84e-4ce6-8736-31b77bb727e5 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.082366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.082968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.083239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.083490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.084268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:32:30.084437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.086067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.098499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1246/2517] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:32:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:32:30.103193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d35829b-f298-4521-98e6-c9a67beb08a9 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.113681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d35829b-f298-4521-98e6-c9a67beb08a9 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.119342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.119950 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.120225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.120494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.121162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.123286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.161990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1272/2518] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:30 2022] POST /identity/v3/projects => generated 377 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:30.166324 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.177010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18eddc85-8a8d-4595-9792-b5c661bb1755 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.236317 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.236922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.237221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.237480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.238122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.240017 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.264054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1247/2519] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:30 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:30.269250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.280386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a4d424-7ccb-431a-aa6b-35079205c163 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.285404 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.286014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.286299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.286670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.288868 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.290589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.300792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1273/2520] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:30.305078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.315455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1850f48a-0d14-415b-91c4-1bc293293069 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.320941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d669eeea-596e-42c2-89c7-3ed4f19ddf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.321563 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d669eeea-596e-42c2-89c7-3ed4f19ddf73 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.321836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d669eeea-596e-42c2-89c7-3ed4f19ddf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.322108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d669eeea-596e-42c2-89c7-3ed4f19ddf73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.322764 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d669eeea-596e-42c2-89c7-3ed4f19ddf73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.324348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d669eeea-596e-42c2-89c7-3ed4f19ddf73 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.335725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1248/2521] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:30.340242 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.352319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a224f58-bb54-47d2-89e5-042f671ca232 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.358512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.359168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.359427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.359675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] PATH_INFO: `/v3/projects/46c07f91d0654fe989092a36705461f1/users/aa7e72b2fb414815b378ab78f01a1ab4/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.375888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=46c07f91d0654fe989092a36705461f1, user_id=aa7e72b2fb414815b378ab78f01a1ab4, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.378635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.393303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1274/2522] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:30 2022] PUT /identity/v3/projects/46c07f91d0654fe989092a36705461f1/users/aa7e72b2fb414815b378ab78f01a1ab4/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:30.404310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5d69d-16da-4789-b86e-b72ec935e4e5 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.404953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5d69d-16da-4789-b86e-b72ec935e4e5 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.405294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5d69d-16da-4789-b86e-b72ec935e4e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.435755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9ed5d69d-16da-4789-b86e-b72ec935e4e5 None None] MFA Rules not processed for user `aa7e72b2fb414815b378ab78f01a1ab4`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:30.436978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9ed5d69d-16da-4789-b86e-b72ec935e4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:30.506447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1249/2523] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:30.511039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.524616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:30.583282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6793d66-c5b0-4585-8465-4ca530f4acf7 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.587960 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.588575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.588851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.589135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.589793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.591961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.621530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1275/2524] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:30 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:32:30.625733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed5d69d-16da-4789-b86e-b72ec935e4e5 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.637382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed5d69d-16da-4789-b86e-b72ec935e4e5 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.652239 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:30.720439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.721066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.721402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.721667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.722322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.724230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.749233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1250/2525] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:30 2022] POST /identity/v3/users => generated 287 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:32:30.753397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.765360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997b4efe-3fcb-4fd4-91fb-baae5ad8816f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.770933 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.772405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.772670 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.772938 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.773690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.775320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.789345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1276/2526] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:30.793603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.805345 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fbedbe-edf1-44fb-93cb-42652cecc776 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.810152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedc83c2-a1bd-486e-936a-ea7f29c0be9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.810763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc83c2-a1bd-486e-936a-ea7f29c0be9a admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.811040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc83c2-a1bd-486e-936a-ea7f29c0be9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.811308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc83c2-a1bd-486e-936a-ea7f29c0be9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.811971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedc83c2-a1bd-486e-936a-ea7f29c0be9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.813615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedc83c2-a1bd-486e-936a-ea7f29c0be9a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.828927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1251/2527] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:30.833175 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:30.844155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99be5790-d9ae-4d8c-9f14-cae898ced1a3 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:30.852425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:30.853106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.853422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.853690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] PATH_INFO: `/v3/projects/15f98c14f24d4443929029ccfffffb41/users/894e9897489248e88c89b09a44592057/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.871729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15f98c14f24d4443929029ccfffffb41, user_id=894e9897489248e88c89b09a44592057, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:30.875606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:30.895625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1277/2528] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:30 2022] PUT /identity/v3/projects/15f98c14f24d4443929029ccfffffb41/users/894e9897489248e88c89b09a44592057/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:30.906426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335891f7-535b-46a7-973a-4a0892e40714 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:30.906719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335891f7-535b-46a7-973a-4a0892e40714 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:30.906982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335891f7-535b-46a7-973a-4a0892e40714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:30.939937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-335891f7-535b-46a7-973a-4a0892e40714 None None] MFA Rules not processed for user `894e9897489248e88c89b09a44592057`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:30.940741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-335891f7-535b-46a7-973a-4a0892e40714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:31.015934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1252/2529] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:30 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:31.021574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:31.092854 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95160c17-60a7-4241-b4bd-dfefae08acf5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:31.101072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:31.101747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.102023 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.102287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.102918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:31.104995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:31.145624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1278/2530] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:31 2022] POST /identity/v3/projects => generated 375 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:31.150980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335891f7-535b-46a7-973a-4a0892e40714 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:31.162831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335891f7-535b-46a7-973a-4a0892e40714 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:31.227343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:31.228417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.228417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.228665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.229136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:31.231074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:31.257915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1253/2531] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:31 2022] POST /identity/v3/users => generated 294 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:31.262914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:31.274010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d40f2b4-8597-48a6-bf8d-725ce89b3287 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:31.279866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:31.280500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.280768 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.281039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.281719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:31.283367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:31.296609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1279/2532] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:31.300965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:31.311343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46199cf1-305d-4251-a079-5ca729011576 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:31.318401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:31.319067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.319338 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.319605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] PATH_INFO: `/v3/projects/129bc6ca31594cfea267998ccd81490d/users/bc3dac4edd7f4cfda55f529a7d0c0b92/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.334081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:31.338357 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=129bc6ca31594cfea267998ccd81490d, user_id=bc3dac4edd7f4cfda55f529a7d0c0b92, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:31.342101 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:31.365525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1254/2533] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:31 2022] PUT /identity/v3/projects/129bc6ca31594cfea267998ccd81490d/users/bc3dac4edd7f4cfda55f529a7d0c0b92/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:31.369739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:31.380451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bd8632-f2bc-4fd4-9da3-d64fe3fe30a2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:31.417207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:31.454095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814bef13-7b7c-4ce6-a8ce-d7771c4131ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:31.455085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814bef13-7b7c-4ce6-a8ce-d7771c4131ec admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.455370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814bef13-7b7c-4ce6-a8ce-d7771c4131ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.455640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814bef13-7b7c-4ce6-a8ce-d7771c4131ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.456298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814bef13-7b7c-4ce6-a8ce-d7771c4131ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:31.457966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814bef13-7b7c-4ce6-a8ce-d7771c4131ec admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:31.472160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1280/2534] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:31.476664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:31.488877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3912cff-a4d3-443e-9929-fe36675dd4e0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:31.494509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:31.495280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.495613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.495909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] PATH_INFO: `/v3/projects/129bc6ca31594cfea267998ccd81490d/users/bc3dac4edd7f4cfda55f529a7d0c0b92/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.498011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] RBAC: Authorizing `identity:create_grant(project_id=129bc6ca31594cfea267998ccd81490d, user_id=bc3dac4edd7f4cfda55f529a7d0c0b92, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:31.500617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:31.512902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1255/2535] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:31 2022] PUT /identity/v3/projects/129bc6ca31594cfea267998ccd81490d/users/bc3dac4edd7f4cfda55f529a7d0c0b92/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:31.525030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.525821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.525821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.559152 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] MFA Rules not processed for user `bc3dac4edd7f4cfda55f529a7d0c0b92`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:31.560493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:31.649710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1281/2536] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:31 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:31.667245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:31.669495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:31.748133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7a6a52-c612-4c22-82fd-ffc8f8a72835 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:31.752247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:31.752869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:31.753163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:31.753438 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:31.847195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:31.848743 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:31.864237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1256/2537] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:33.535978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:33.537679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:33.551369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e388edd-2168-48ca-862c-3e7b61f47032 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:33.555569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:33.556174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:33.556447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:33.556715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:33.623826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:33.629380 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:33.644095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1282/2538] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:34.316509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:34.329833 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a93336-cfce-4acc-ab92-3d7bd4f5aaf9 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:34.336262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8cbdf4-3289-4f50-9efe-cdcdc7b3ab19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:34.336921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8cbdf4-3289-4f50-9efe-cdcdc7b3ab19 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:34.337221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8cbdf4-3289-4f50-9efe-cdcdc7b3ab19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:34.337490 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8cbdf4-3289-4f50-9efe-cdcdc7b3ab19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:34.349556 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8cbdf4-3289-4f50-9efe-cdcdc7b3ab19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:34.352477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8cbdf4-3289-4f50-9efe-cdcdc7b3ab19 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:34.365821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1257/2539] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:34 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:38.287162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:38.357990 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:38.363561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150503ba-92a1-4b89-be8d-2961e0817639 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:38.370485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:38.371130 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:38.371409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:38.371683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] PATH_INFO: `/v3/users/aa7e72b2fb414815b378ab78f01a1ab4` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:38.374846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa7e72b2fb414815b378ab78f01a1ab4)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:38.376836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:38.437137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa7e72b2fb414815b378ab78f01a1ab4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:38.452079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa7e72b2fb414815b378ab78f01a1ab4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:38.452784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa7e72b2fb414815b378ab78f01a1ab4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:38.464995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa7e72b2fb414815b378ab78f01a1ab4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:38.475166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa7e72b2fb414815b378ab78f01a1ab4'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:38.484676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1283/2540] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:38 2022] DELETE /identity/v3/users/aa7e72b2fb414815b378ab78f01a1ab4 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:38.489594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:38.490134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:38.490134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:38.514657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:38.515767 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:38.581628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1258/2541] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:38.593079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:38.601010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:38.662596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8452d0d-dffb-4631-8006-54592d7e02e2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:38.666633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:38.667243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:38.667518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:38.667787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:38.677752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:38.679035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:38.692503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1284/2542] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:38.976505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:38.978055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:38.991187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8ad63d-2b75-4349-8a08-05ebfe1a86dc None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:38.998978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:38.999636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:38.999901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:39.000171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] PATH_INFO: `/v3/projects/46c07f91d0654fe989092a36705461f1` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:39.001368 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] RBAC: Authorizing `identity:delete_project(project_id=46c07f91d0654fe989092a36705461f1)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:39.003268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:39.020545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:39.055982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46c07f91d0654fe989092a36705461f1', 'request_id': 'req-0e1c4b9d-d408-4281-b995-15f67af71c7b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.066090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46c07f91d0654fe989092a36705461f1', 'request_id': 'req-0e1c4b9d-d408-4281-b995-15f67af71c7b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.074103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46c07f91d0654fe989092a36705461f1', 'request_id': 'req-0e1c4b9d-d408-4281-b995-15f67af71c7b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.086213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] The token cache is being invalidate because project 46c07f91d0654fe989092a36705461f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:39.086497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.087216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.088755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1259/2543] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:38 2022] DELETE /identity/v3/projects/46c07f91d0654fe989092a36705461f1 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:39.092816 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:39.103260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770196d5-d133-4d4c-ab3c-0bdd1c94219f service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:39.172110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:39.173036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:39.173582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:39.173973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] PATH_INFO: `/v3/users/894e9897489248e88c89b09a44592057` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:39.175245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] RBAC: Authorizing `identity:delete_user(user_id=894e9897489248e88c89b09a44592057)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:39.177283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:39.209901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:39.256664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '894e9897489248e88c89b09a44592057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.271665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '894e9897489248e88c89b09a44592057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.272432 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '894e9897489248e88c89b09a44592057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.282036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '894e9897489248e88c89b09a44592057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.289614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '894e9897489248e88c89b09a44592057'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.296097 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1285/2544] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:39 2022] DELETE /identity/v3/users/894e9897489248e88c89b09a44592057 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:39.578918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:39.580455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:39.594585 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1c4b9d-d408-4281-b995-15f67af71c7b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:39.655354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:39.655980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:39.656251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:39.656515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] PATH_INFO: `/v3/projects/15f98c14f24d4443929029ccfffffb41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:39.657670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15f98c14f24d4443929029ccfffffb41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:39.659593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:39.712538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15f98c14f24d4443929029ccfffffb41', 'request_id': 'req-0ae9a9ca-4453-496f-8b78-fabf15ed0229'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.722588 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15f98c14f24d4443929029ccfffffb41', 'request_id': 'req-0ae9a9ca-4453-496f-8b78-fabf15ed0229'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.728971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15f98c14f24d4443929029ccfffffb41', 'request_id': 'req-0ae9a9ca-4453-496f-8b78-fabf15ed0229'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.742349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] The token cache is being invalidate because project 15f98c14f24d4443929029ccfffffb41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:39.742630 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.743292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.744802 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1260/2545] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:39 2022] DELETE /identity/v3/projects/15f98c14f24d4443929029ccfffffb41 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:39.749317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:39.750846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:39.765088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723d12d8-bc19-49b9-a289-27220378bb46 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:39.827496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:39.828057 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:39.828325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:39.828594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] PATH_INFO: `/v3/users/bc3dac4edd7f4cfda55f529a7d0c0b92` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:39.829738 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc3dac4edd7f4cfda55f529a7d0c0b92)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:39.831628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:39.878275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc3dac4edd7f4cfda55f529a7d0c0b92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.890484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc3dac4edd7f4cfda55f529a7d0c0b92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.891260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc3dac4edd7f4cfda55f529a7d0c0b92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.899338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc3dac4edd7f4cfda55f529a7d0c0b92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.907134 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ca36564f-fe22-4581-89cc-b1bb2874a023 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc3dac4edd7f4cfda55f529a7d0c0b92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:39.913783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1286/2546] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:39 2022] DELETE /identity/v3/users/bc3dac4edd7f4cfda55f529a7d0c0b92 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:32:40.206175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:40.207710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:40.222571 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae9a9ca-4453-496f-8b78-fabf15ed0229 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:40.255349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:40.282302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:40.282956 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.283211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.283489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] PATH_INFO: `/v3/projects/129bc6ca31594cfea267998ccd81490d` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.284612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=129bc6ca31594cfea267998ccd81490d)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:40.286558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:40.336033 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '129bc6ca31594cfea267998ccd81490d', 'request_id': 'req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:40.344535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '129bc6ca31594cfea267998ccd81490d', 'request_id': 'req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:40.351287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '129bc6ca31594cfea267998ccd81490d', 'request_id': 'req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:40.362098 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] The token cache is being invalidate because project 129bc6ca31594cfea267998ccd81490d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:40.362377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:40.363029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3b20afc7-d305-4ea8-adbc-89aafcf7c8a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:40.364616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1261/2547] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:40 2022] DELETE /identity/v3/projects/129bc6ca31594cfea267998ccd81490d => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:40.370745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4714db7-f097-47ab-a14c-31965ed1af96 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.371233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4714db7-f097-47ab-a14c-31965ed1af96 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.371339 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4714db7-f097-47ab-a14c-31965ed1af96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.398386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c4714db7-f097-47ab-a14c-31965ed1af96 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:40.399622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c4714db7-f097-47ab-a14c-31965ed1af96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:40.410978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:40.479081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1287/2548] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:40.487171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe79eca-f751-4229-9207-02ed3d3f4cbb None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.487448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe79eca-f751-4229-9207-02ed3d3f4cbb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.487715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe79eca-f751-4229-9207-02ed3d3f4cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.508975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-0fe79eca-f751-4229-9207-02ed3d3f4cbb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:40.510233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0fe79eca-f751-4229-9207-02ed3d3f4cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:40.519409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1262/2549] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:40.524083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4714db7-f097-47ab-a14c-31965ed1af96 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:40.534719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4714db7-f097-47ab-a14c-31965ed1af96 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:40.540326 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:40.540985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.541297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.541561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.542236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:32:40.542537 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:40.544291 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:40.558114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1288/2550] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:32:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:32:40.562838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe79eca-f751-4229-9207-02ed3d3f4cbb None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:40.572768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe79eca-f751-4229-9207-02ed3d3f4cbb None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:40.578003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:40.578601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.578872 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.579131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.579714 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:40.581671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:40.612942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1263/2551] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:40 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:40.617270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:40.628142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf750c4-f8ee-4e3e-b144-716fa54301d1 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:40.690831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:40.691440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.691845 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.692124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.692792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:40.694720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:40.717106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1289/2552] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:40 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:32:40.721448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:40.732819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ac8c32-4016-4cbc-8bd0-55e24f3296bc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:40.738590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:40.739183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.739474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.739735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.740375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:40.742163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:40.753268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1264/2553] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:40.758240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:40.769343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c9046-8889-4568-96d5-6656d0623f64 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:40.774411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cadcfe-d233-42c4-8396-2f9422058c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:40.775012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cadcfe-d233-42c4-8396-2f9422058c1d admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.775294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cadcfe-d233-42c4-8396-2f9422058c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.775563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cadcfe-d233-42c4-8396-2f9422058c1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.776203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cadcfe-d233-42c4-8396-2f9422058c1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:40.777855 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cadcfe-d233-42c4-8396-2f9422058c1d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:40.790061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1290/2554] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:40.795222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:40.805646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fc6699-9562-48e5-a0b5-8f8ca30e785b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:40.811431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:40.812108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.812385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.812658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] PATH_INFO: `/v3/projects/b366f7d31c704c18898a02b9fdceac06/users/3d852c1a37674adcaeb319831fd70ff4/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.828252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] RBAC: Authorizing `identity:create_grant(project_id=b366f7d31c704c18898a02b9fdceac06, user_id=3d852c1a37674adcaeb319831fd70ff4, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:40.830935 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:40.847479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1265/2555] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:40 2022] PUT /identity/v3/projects/b366f7d31c704c18898a02b9fdceac06/users/3d852c1a37674adcaeb319831fd70ff4/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:40.917286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6400cd-1953-4c81-8385-d2b4b41c2845 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:40.917519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6400cd-1953-4c81-8385-d2b4b41c2845 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:40.917730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6400cd-1953-4c81-8385-d2b4b41c2845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:40.955699 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-ca6400cd-1953-4c81-8385-d2b4b41c2845 None None] MFA Rules not processed for user `3d852c1a37674adcaeb319831fd70ff4`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:40.956896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ca6400cd-1953-4c81-8385-d2b4b41c2845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:41.022117 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1291/2556] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:40 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:41.027198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.096197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424846b3-f581-4970-b375-47a1583eadab admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.104138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.105091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.105471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.105812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.106801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.109510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.114619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:41.139494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1266/2557] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:41 2022] POST /identity/v3/projects => generated 383 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:41.143746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6400cd-1953-4c81-8385-d2b4b41c2845 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.154687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6400cd-1953-4c81-8385-d2b4b41c2845 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.163582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:41.223626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.224144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.224423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.224689 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.225524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.227449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.249441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1292/2558] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:41 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:32:41.253575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.265474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd22985-d3b2-44af-83a4-a880e4f95e59 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.271190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.271799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.272067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.272330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.273000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.274642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.285848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1267/2559] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:41.290420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.300869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12457974-b341-4c63-9250-85c38cca8b5a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.307389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b3e1c4-4f5b-40b5-a9eb-a44e9b0be852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.307989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b3e1c4-4f5b-40b5-a9eb-a44e9b0be852 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.308261 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b3e1c4-4f5b-40b5-a9eb-a44e9b0be852 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.308525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b3e1c4-4f5b-40b5-a9eb-a44e9b0be852 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.309199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b3e1c4-4f5b-40b5-a9eb-a44e9b0be852 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.310852 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b3e1c4-4f5b-40b5-a9eb-a44e9b0be852 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.321278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1293/2560] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:41.326274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.337054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09974a7f-02a4-4904-9e28-5a98dba07f7b admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.344389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.345208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.345208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.345622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] PATH_INFO: `/v3/projects/4fab7d3da15740c4a0d2f7fe3b014d51/users/f2ffc46b62844a87b033b0781173364e/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.362934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4fab7d3da15740c4a0d2f7fe3b014d51, user_id=f2ffc46b62844a87b033b0781173364e, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.364919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.380596 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1268/2561] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:41 2022] PUT /identity/v3/projects/4fab7d3da15740c4a0d2f7fe3b014d51/users/f2ffc46b62844a87b033b0781173364e/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:41.397492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ac09c-531a-4a29-8e4a-fa2353c79082 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.398009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ac09c-531a-4a29-8e4a-fa2353c79082 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.398009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ac09c-531a-4a29-8e4a-fa2353c79082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.436773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-515ac09c-531a-4a29-8e4a-fa2353c79082 None None] MFA Rules not processed for user `f2ffc46b62844a87b033b0781173364e`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:41.438576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-515ac09c-531a-4a29-8e4a-fa2353c79082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:41.510527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1294/2562] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:41.515212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.582936 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61edcf38-36c5-4549-9e8e-4cab5d838859 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.589479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.590088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.590359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.590646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.591384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.593295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.623892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1269/2563] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:41 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:41.629482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515ac09c-531a-4a29-8e4a-fa2353c79082 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.642383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515ac09c-531a-4a29-8e4a-fa2353c79082 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.714338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.715127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.715503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.715859 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.717692 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.719698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.744754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1295/2564] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:41 2022] POST /identity/v3/users => generated 299 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:41.749905 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.760192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a567485-06a1-4e92-b4ba-90e1fcd625a8 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.765869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.766390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.766698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.766973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.767680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.769354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.784474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1270/2565] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:41.788418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.798695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67743554-2ef7-4350-b89c-fd88a1bd4aaa admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.803563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.804297 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.804571 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.804881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] PATH_INFO: `/v3/projects/e69f8a796636461bb7e4e693b62eb340/users/38f23afe744648f9930f59c215ba746c/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.808894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:41.823914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e69f8a796636461bb7e4e693b62eb340, user_id=38f23afe744648f9930f59c215ba746c, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.826720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.850037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1296/2566] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:41 2022] PUT /identity/v3/projects/e69f8a796636461bb7e4e693b62eb340/users/38f23afe744648f9930f59c215ba746c/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:41.855369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.871185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f14a7e4-2f88-403e-8f21-41894dc3799d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.929236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d18968-bf52-4b3c-8d8b-347e5577e66e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.929841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d18968-bf52-4b3c-8d8b-347e5577e66e admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.930100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d18968-bf52-4b3c-8d8b-347e5577e66e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.930365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d18968-bf52-4b3c-8d8b-347e5577e66e admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.931010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d18968-bf52-4b3c-8d8b-347e5577e66e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.932591 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d18968-bf52-4b3c-8d8b-347e5577e66e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.942551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1271/2567] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:41 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:41.947037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:41.960811 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d6cad-a881-4fe0-bd68-618a4311c34b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:41.965589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:41.966245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.966694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.966925 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] PATH_INFO: `/v3/projects/e69f8a796636461bb7e4e693b62eb340/users/38f23afe744648f9930f59c215ba746c/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:41.968989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e69f8a796636461bb7e4e693b62eb340, user_id=38f23afe744648f9930f59c215ba746c, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:41.971983 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:41.983224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1297/2568] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:41 2022] PUT /identity/v3/projects/e69f8a796636461bb7e4e693b62eb340/users/38f23afe744648f9930f59c215ba746c/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:32:41.993806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:41.994089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:41.994477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:42.028908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] MFA Rules not processed for user `38f23afe744648f9930f59c215ba746c`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:42.030364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:42.039953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:42.124181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1272/2569] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:41 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:42.136407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:42.139174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:42.202588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb8cd9-d408-431e-a97f-22feccc0e888 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:42.206411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:42.207019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:42.207293 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:42.207560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:42.272946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:42.274401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:42.284478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1298/2570] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:42 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:43.819561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:43.821160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:43.834541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1a8bf9-aa82-40ac-86cc-8b78de9ade9e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:43.840095 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:43.840962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:43.841319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:43.841607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:43.902168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:43.904373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:43.913473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1273/2571] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:44.504672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:44.514179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f861c3-be75-4c4a-bf28-a9ce378eca97 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:44.519107 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221faf2f-a621-42ce-aff0-2cf7ba59ef90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:44.520106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221faf2f-a621-42ce-aff0-2cf7ba59ef90 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:44.520464 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221faf2f-a621-42ce-aff0-2cf7ba59ef90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:44.520706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221faf2f-a621-42ce-aff0-2cf7ba59ef90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:44.530892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221faf2f-a621-42ce-aff0-2cf7ba59ef90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:44.532319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221faf2f-a621-42ce-aff0-2cf7ba59ef90 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:44.541922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1299/2572] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:44 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:48.529636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:48.600372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb57a56-1a75-424d-8e02-925ea7db2cf5 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:48.606040 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:48.606631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:48.607016 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:48.607847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] PATH_INFO: `/v3/users/3d852c1a37674adcaeb319831fd70ff4` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:48.609042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d852c1a37674adcaeb319831fd70ff4)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:48.611089 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:48.662008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:48.664813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d852c1a37674adcaeb319831fd70ff4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:48.674218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d852c1a37674adcaeb319831fd70ff4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:48.684720 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d852c1a37674adcaeb319831fd70ff4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:48.687237 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d852c1a37674adcaeb319831fd70ff4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:48.691052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d852c1a37674adcaeb319831fd70ff4'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:48.711505 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1274/2573] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:48 2022] DELETE /identity/v3/users/3d852c1a37674adcaeb319831fd70ff4 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:48.717598 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:48.717977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:48.718226 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:48.738568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:48.740207 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:48.803620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1300/2574] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:48.814035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:48.815589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:48.876515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e748d1-dc06-4efd-8676-7c48d5c5921e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:48.884717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:48.885386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:48.885713 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:48.885981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:48.896468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:48.897930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:48.907923 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1275/2575] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:49.208067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:49.209626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:49.224662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bbe0e2-04cd-4996-9370-1274e0d20fb0 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:49.230233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:49.230880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:49.231157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:49.231422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] PATH_INFO: `/v3/projects/b366f7d31c704c18898a02b9fdceac06` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:49.232932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b366f7d31c704c18898a02b9fdceac06)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:49.235029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:49.244480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:49.293903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b366f7d31c704c18898a02b9fdceac06', 'request_id': 'req-9164b9d5-ddbd-49fb-a40e-6f098eeff912'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.303618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b366f7d31c704c18898a02b9fdceac06', 'request_id': 'req-9164b9d5-ddbd-49fb-a40e-6f098eeff912'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.313034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b366f7d31c704c18898a02b9fdceac06', 'request_id': 'req-9164b9d5-ddbd-49fb-a40e-6f098eeff912'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.331366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] The token cache is being invalidate because project b366f7d31c704c18898a02b9fdceac06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:49.331611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.332313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.333815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1301/2576] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:49 2022] DELETE /identity/v3/projects/b366f7d31c704c18898a02b9fdceac06 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:49.338641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:49.349039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b8d4ae-df40-4bbe-ac00-41832075f523 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:49.410979 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:49.411616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:49.411885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:49.412150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] PATH_INFO: `/v3/users/f2ffc46b62844a87b033b0781173364e` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:49.413437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2ffc46b62844a87b033b0781173364e)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:49.415359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:49.471062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2ffc46b62844a87b033b0781173364e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.478738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2ffc46b62844a87b033b0781173364e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.488271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2ffc46b62844a87b033b0781173364e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.489008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2ffc46b62844a87b033b0781173364e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.489791 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2ffc46b62844a87b033b0781173364e'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.495499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:49.511778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1276/2577] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:49 2022] DELETE /identity/v3/users/f2ffc46b62844a87b033b0781173364e => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:49.807012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:49.809209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:49.824497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9164b9d5-ddbd-49fb-a40e-6f098eeff912 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:49.885867 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:49.886504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:49.886786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:49.887055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] PATH_INFO: `/v3/projects/4fab7d3da15740c4a0d2f7fe3b014d51` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:49.890193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4fab7d3da15740c4a0d2f7fe3b014d51)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:49.890741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:49.954340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4fab7d3da15740c4a0d2f7fe3b014d51', 'request_id': 'req-277ca753-eca5-44d5-bca5-37f01ef9e199'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.964156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4fab7d3da15740c4a0d2f7fe3b014d51', 'request_id': 'req-277ca753-eca5-44d5-bca5-37f01ef9e199'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.974601 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4fab7d3da15740c4a0d2f7fe3b014d51', 'request_id': 'req-277ca753-eca5-44d5-bca5-37f01ef9e199'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.995314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] The token cache is being invalidate because project 4fab7d3da15740c4a0d2f7fe3b014d51 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:49.995791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.996758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:49.999971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1302/2578] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:49 2022] DELETE /identity/v3/projects/4fab7d3da15740c4a0d2f7fe3b014d51 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:50.004885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:50.006465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:50.020990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c980a89-fa71-41f7-b313-086b2507a4f1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:50.079456 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:50.080102 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:50.080372 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:50.080624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] PATH_INFO: `/v3/users/38f23afe744648f9930f59c215ba746c` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:50.081792 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=38f23afe744648f9930f59c215ba746c)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:50.083694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:50.135246 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38f23afe744648f9930f59c215ba746c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.142220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38f23afe744648f9930f59c215ba746c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.150939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38f23afe744648f9930f59c215ba746c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.151558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38f23afe744648f9930f59c215ba746c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.152161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-085b19b0-8f2d-4af4-a2b5-14e2ce871c9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38f23afe744648f9930f59c215ba746c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.170290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1277/2579] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:32:50 2022] DELETE /identity/v3/users/38f23afe744648f9930f59c215ba746c => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:50.467970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:50.469793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:50.485904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ca753-eca5-44d5-bca5-37f01ef9e199 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:50.515408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:50.556465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:50.557022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:50.557324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:50.557590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] PATH_INFO: `/v3/projects/e69f8a796636461bb7e4e693b62eb340` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:50.558727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e69f8a796636461bb7e4e693b62eb340)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:50.561053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:50.615024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e69f8a796636461bb7e4e693b62eb340', 'request_id': 'req-814d451a-1e10-49a0-8b51-0a361a646bcb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.623949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e69f8a796636461bb7e4e693b62eb340', 'request_id': 'req-814d451a-1e10-49a0-8b51-0a361a646bcb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.630373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e69f8a796636461bb7e4e693b62eb340', 'request_id': 'req-814d451a-1e10-49a0-8b51-0a361a646bcb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.644460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] The token cache is being invalidate because project e69f8a796636461bb7e4e693b62eb340 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:32:50.644918 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.645707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-814d451a-1e10-49a0-8b51-0a361a646bcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:32:50.647457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1303/2580] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:32:50 2022] DELETE /identity/v3/projects/e69f8a796636461bb7e4e693b62eb340 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:32:50.653648 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b10911-0eac-436f-9ae8-630c2559f895 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:50.654172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b10911-0eac-436f-9ae8-630c2559f895 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:50.654287 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b10911-0eac-436f-9ae8-630c2559f895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:50.677900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-19b10911-0eac-436f-9ae8-630c2559f895 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:50.679118 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-19b10911-0eac-436f-9ae8-630c2559f895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:50.746875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1278/2581] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:50.755141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66539d42-36e0-4bee-8e61-0033994a3255 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:50.755440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66539d42-36e0-4bee-8e61-0033994a3255 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:50.755706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66539d42-36e0-4bee-8e61-0033994a3255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:50.776942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-66539d42-36e0-4bee-8e61-0033994a3255 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:50.778232 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-66539d42-36e0-4bee-8e61-0033994a3255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:50.787029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1304/2582] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:32:50.791750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b10911-0eac-436f-9ae8-630c2559f895 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:50.798573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:50.801920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b10911-0eac-436f-9ae8-630c2559f895 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:50.808503 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:50.809509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:50.809982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:50.810433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:50.811303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:32:50.811606 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:50.813290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:50.825178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1279/2583] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:32:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:32:50.830722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66539d42-36e0-4bee-8e61-0033994a3255 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:50.843632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66539d42-36e0-4bee-8e61-0033994a3255 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:50.848793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:50.849456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:50.849733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:50.849997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:50.850632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:50.852539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:50.883799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1305/2584] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:32:50 2022] POST /identity/v3/projects => generated 367 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:50.888348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:50.898819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc64ba4a-cfd5-40e5-be79-e4e43b704d70 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:50.961850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:50.962590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:50.962863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:50.963123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:50.963762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:50.965799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:50.991535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1280/2585] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:32:50 2022] POST /identity/v3/users => generated 284 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:32:50.996194 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:51.010608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f51987-21d1-40e2-86f4-806f687160a2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:51.017353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:51.017982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:51.018254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:51.018527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:51.019178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:51.020798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:51.032889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1306/2586] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:50 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:51.037214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:51.048045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ab61f4-68a8-4a24-a58b-cb0d2fae6a9d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:51.053700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749545d1-3f11-4514-b427-2d72fba9bbd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:51.054422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749545d1-3f11-4514-b427-2d72fba9bbd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:51.054693 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749545d1-3f11-4514-b427-2d72fba9bbd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:51.054946 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749545d1-3f11-4514-b427-2d72fba9bbd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:51.055612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749545d1-3f11-4514-b427-2d72fba9bbd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:51.057286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749545d1-3f11-4514-b427-2d72fba9bbd7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:51.068876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1281/2587] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:32:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:32:51.072996 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:51.083620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d136d2-9507-400c-b3bf-2b503cd556c8 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:51.088709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:51.089485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:51.089771 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:51.090037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] PATH_INFO: `/v3/projects/2401832e5c2f45b8aad698537190a215/users/ffd439aa4e91466cbfcaf7b2473bc192/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:51.107429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2401832e5c2f45b8aad698537190a215, user_id=ffd439aa4e91466cbfcaf7b2473bc192, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:51.110721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:51.125458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1307/2588] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:32:51 2022] PUT /identity/v3/projects/2401832e5c2f45b8aad698537190a215/users/ffd439aa4e91466cbfcaf7b2473bc192/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:32:51.136084 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b097b-146f-492b-b2b1-34a47a9ff5e7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:51.136327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b097b-146f-492b-b2b1-34a47a9ff5e7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:51.136580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b097b-146f-492b-b2b1-34a47a9ff5e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:51.167382 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-887b097b-146f-492b-b2b1-34a47a9ff5e7 None None] MFA Rules not processed for user `ffd439aa4e91466cbfcaf7b2473bc192`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:32:51.168610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-887b097b-146f-492b-b2b1-34a47a9ff5e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:51.238862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1282/2589] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:32:51 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:32:51.252829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:51.254826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:51.320765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6099342b-d8cf-4fed-996f-4d55832cd698 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:51.325252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:51.326096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:51.327031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:51.327031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:51.334944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:51.337947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:51.340093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:51.352139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1308/2590] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:51 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:53.377233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887b097b-146f-492b-b2b1-34a47a9ff5e7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:53.390608 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887b097b-146f-492b-b2b1-34a47a9ff5e7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:53.395384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f83a8d2-58b4-4a12-ae82-8039ddfeaf3a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:53.396003 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83a8d2-58b4-4a12-ae82-8039ddfeaf3a service glance] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:53.396275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83a8d2-58b4-4a12-ae82-8039ddfeaf3a service glance] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:53.396537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83a8d2-58b4-4a12-ae82-8039ddfeaf3a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:53.398546 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5f83a8d2-58b4-4a12-ae82-8039ddfeaf3a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:53.469645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:32:53.491932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f83a8d2-58b4-4a12-ae82-8039ddfeaf3a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:53.493410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f83a8d2-58b4-4a12-ae82-8039ddfeaf3a service glance] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:53.505611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1283/2591] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:32:53.683082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:32:53.684795 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:53.753053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb174fdc-1c81-4603-841b-7720f2852506 service glance] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:32:53.757461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:32:53.758071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:32:53.758344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:32:53.758610 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:32:53.760578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:32:53.776469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:32:53.778065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:32:53.790500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1309/2592] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:32:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:33:03.499653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=7 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:33:03.785160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=28 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:33:16.868526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6434b-9dce-4e1e-b020-21917bb08dd1 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:16.869024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6434b-9dce-4e1e-b020-21917bb08dd1 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:16.869163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6434b-9dce-4e1e-b020-21917bb08dd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:16.940341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-4ce6434b-9dce-4e1e-b020-21917bb08dd1 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:33:16.941620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-4ce6434b-9dce-4e1e-b020-21917bb08dd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:17.017995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1284/2593] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:33:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:33:17.069592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:17.071196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:17.164952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa068ab-a689-4b12-b04a-fa664c60092f service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:17.170080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a40cef-dce2-47f0-89d0-50b8e6aba266 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:17.170682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a40cef-dce2-47f0-89d0-50b8e6aba266 service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:17.170956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a40cef-dce2-47f0-89d0-50b8e6aba266 service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:17.171223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a40cef-dce2-47f0-89d0-50b8e6aba266 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:17.179824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a40cef-dce2-47f0-89d0-50b8e6aba266 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:17.181455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a40cef-dce2-47f0-89d0-50b8e6aba266 service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:17.189558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1310/2594] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:33:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:33:27.013368 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=44 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:33:27.186156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=41 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:33:35.716575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce6434b-9dce-4e1e-b020-21917bb08dd1 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:35.727545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce6434b-9dce-4e1e-b020-21917bb08dd1 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:35.732447 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b722f1b-4ecd-43e7-b122-5445d1d8b247 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:35.733025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b722f1b-4ecd-43e7-b122-5445d1d8b247 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:35.733349 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b722f1b-4ecd-43e7-b122-5445d1d8b247 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:35.733611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b722f1b-4ecd-43e7-b122-5445d1d8b247 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:35.735682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8b722f1b-4ecd-43e7-b122-5445d1d8b247 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:35.746851 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b722f1b-4ecd-43e7-b122-5445d1d8b247 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:35.748427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b722f1b-4ecd-43e7-b122-5445d1d8b247 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:35.757011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1285/2595] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:33:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:33:40.788927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b644180a-066f-40b3-be1d-9de675c99d5f None None] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:40.789551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b644180a-066f-40b3-be1d-9de675c99d5f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:40.789551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b644180a-066f-40b3-be1d-9de675c99d5f None None] PATH_INFO: `/` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:40.790891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1311/2596] 10.222.0.52 () {54 vars in 921 bytes} [Sat Aug 27 02:33:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 27 02:33:40.795962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:40.796444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:40.796689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:40.838444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] MFA Rules not processed for user `c83f0d0e2e8047a98fbe114ceb15c6e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:33:40.839666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:40.851640 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1286/2597] 10.222.0.52 () {58 vars in 1018 bytes} [Sat Aug 27 02:33:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:33:40.863649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b644180a-066f-40b3-be1d-9de675c99d5f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:40.875660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b644180a-066f-40b3-be1d-9de675c99d5f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:40.880296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:40.880910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:40.881228 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:40.881499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:40.892582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:40.894037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:40.905707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1312/2598] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:33:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:33:42.142211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:42.143818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:42.158332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:42.222899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac27a8b6-ee39-443f-b1e1-9e495ef3d269 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:42.227614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a54be8-8d43-4433-8167-597f0d682a86 service glance] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:42.228766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf2e2d-54f6-4bba-8162-e64ce5fc23aa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:42.229477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf2e2d-54f6-4bba-8162-e64ce5fc23aa service placement] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:42.229809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf2e2d-54f6-4bba-8162-e64ce5fc23aa service placement] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:42.230141 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf2e2d-54f6-4bba-8162-e64ce5fc23aa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:42.232025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d2bf2e2d-54f6-4bba-8162-e64ce5fc23aa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:42.232498 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:42.233179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:42.233454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:42.233723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] PATH_INFO: `/v3/users/ffd439aa4e91466cbfcaf7b2473bc192` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:42.234902 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffd439aa4e91466cbfcaf7b2473bc192)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:42.236830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:42.245330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf2e2d-54f6-4bba-8162-e64ce5fc23aa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:42.247000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf2e2d-54f6-4bba-8162-e64ce5fc23aa service placement] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:42.260877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1287/2599] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:33:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:33:42.294629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffd439aa4e91466cbfcaf7b2473bc192'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.309393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffd439aa4e91466cbfcaf7b2473bc192'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.310218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffd439aa4e91466cbfcaf7b2473bc192'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.319798 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffd439aa4e91466cbfcaf7b2473bc192'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.328119 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffd439aa4e91466cbfcaf7b2473bc192'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.334269 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1313/2600] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:33:42 2022] DELETE /identity/v3/users/ffd439aa4e91466cbfcaf7b2473bc192 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:33:42.338939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:42.339293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:42.339698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:42.360294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:33:42.361568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:42.438741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1288/2601] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:33:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:33:42.449460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:42.451043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:42.519413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe277b-9962-430c-8c9b-0b00354d7d3a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:42.524822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fb9068-820f-4536-b2bb-0c95cc83249d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:42.525466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fb9068-820f-4536-b2bb-0c95cc83249d service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:42.525742 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fb9068-820f-4536-b2bb-0c95cc83249d service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:42.526007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fb9068-820f-4536-b2bb-0c95cc83249d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:42.535786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fb9068-820f-4536-b2bb-0c95cc83249d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:42.537932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fb9068-820f-4536-b2bb-0c95cc83249d service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:42.547779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1314/2602] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:33:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:33:42.827684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:42.829219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:42.841862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40f54f-d3d5-4998-9551-bae436862458 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:42.858413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:42.859071 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:42.859336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:42.859594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] PATH_INFO: `/v3/projects/2401832e5c2f45b8aad698537190a215` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:42.860893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] RBAC: Authorizing `identity:delete_project(project_id=2401832e5c2f45b8aad698537190a215)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:42.864610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:42.921842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2401832e5c2f45b8aad698537190a215', 'request_id': 'req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.924479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:33:42.933548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2401832e5c2f45b8aad698537190a215', 'request_id': 'req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.941787 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2401832e5c2f45b8aad698537190a215', 'request_id': 'req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.953054 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] The token cache is being invalidate because project 2401832e5c2f45b8aad698537190a215 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:33:42.953455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.954103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:33:42.955604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1289/2603] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:33:42 2022] DELETE /identity/v3/projects/2401832e5c2f45b8aad698537190a215 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:33:47.075892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:47.076377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:47.076466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:47.097340 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:33:47.098765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:47.103379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:33:47.158837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1315/2604] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:33:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:33:47.173189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:33:47.175599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:33:47.255160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c0f310-dd95-4ad8-afc3-2ce81a4aa3be admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:33:47.260830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6317245e-7821-434b-a8dc-e7ed7009390c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:33:47.261486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6317245e-7821-434b-a8dc-e7ed7009390c service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:33:47.261756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6317245e-7821-434b-a8dc-e7ed7009390c service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:33:47.262029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6317245e-7821-434b-a8dc-e7ed7009390c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:33:47.270418 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6317245e-7821-434b-a8dc-e7ed7009390c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:33:47.272232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6317245e-7821-434b-a8dc-e7ed7009390c service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:33:47.280291 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1290/2605] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:33:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:33:57.151075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=21 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:33:57.277396 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=3,keystone:SELECT=38 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:04.435287 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:04.436959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:04.524998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ace333-07ad-487f-8b41-61c7c72a4844 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:04.530628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:04.531163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:04.531433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:04.531697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:04.533499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:04.545959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:04.547539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:04.558310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1316/2606] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:06.409074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:06.409594 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:06.409700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:06.433658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:06.434867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:06.443285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1291/2607] 10.222.0.52 () {58 vars in 967 bytes} [Sat Aug 27 02:34:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:06.456889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:06.474760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c968b7-decd-4000-9430-332a3128d74a service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:06.481324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:06.481921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:06.482190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:06.482468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:06.491102 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:06.492726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:06.501267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1317/2608] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:34:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:34:16.428516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:16.497563 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=40 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:34.654612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:34.656657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:34.716807 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d13f067-5611-48ca-a1a4-a98dcfe01214 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:34.720962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:34.721636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:34.721904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:34.722165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:34.724043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:34.799225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:34.800690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:34.810984 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1292/2609] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:38.332640 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:38.335280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:38.352618 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0db41e-ff8c-4280-9118-5cfde06a43b7 service swift] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:38.359012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:38.359661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:38.359937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:38.360206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] PATH_INFO: `/v3/users/26a86351583d49d090b3db09f127d49f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:38.375871 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26a86351583d49d090b3db09f127d49f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:38.377793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:38.435476 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26a86351583d49d090b3db09f127d49f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.449695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26a86351583d49d090b3db09f127d49f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.450504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26a86351583d49d090b3db09f127d49f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.458830 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26a86351583d49d090b3db09f127d49f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.466295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-947e419a-f08b-4d2e-948c-269ac8ca8733 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26a86351583d49d090b3db09f127d49f'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.472362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1318/2610] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:34:38 2022] DELETE /identity/v3/users/26a86351583d49d090b3db09f127d49f => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:38.673415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:38.684334 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c1ee93-a447-48f1-9cb9-03f62f9d4d1f service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:38.749735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:38.750768 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:38.751048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:38.751320 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] PATH_INFO: `/v3/projects/5d5ec8450c0941d18d63558baab54737` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:38.764093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d5ec8450c0941d18d63558baab54737)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:38.766448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:38.820537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:38.825793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d5ec8450c0941d18d63558baab54737', 'request_id': 'req-e745889b-b106-44ca-8720-a218f98df0f7'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.836680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d5ec8450c0941d18d63558baab54737', 'request_id': 'req-e745889b-b106-44ca-8720-a218f98df0f7'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.845386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d5ec8450c0941d18d63558baab54737', 'request_id': 'req-e745889b-b106-44ca-8720-a218f98df0f7'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.856651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] The token cache is being invalidate because project 5d5ec8450c0941d18d63558baab54737 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:34:38.856942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.857706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e745889b-b106-44ca-8720-a218f98df0f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:38.859214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1293/2611] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:34:38 2022] DELETE /identity/v3/projects/5d5ec8450c0941d18d63558baab54737 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:38.864384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a72cf8c-6e41-4181-8be7-e2b982087762 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:38.864682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a72cf8c-6e41-4181-8be7-e2b982087762 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:38.864955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a72cf8c-6e41-4181-8be7-e2b982087762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:38.885643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5a72cf8c-6e41-4181-8be7-e2b982087762 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:38.886876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5a72cf8c-6e41-4181-8be7-e2b982087762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:38.952503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1319/2612] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:38.961333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0921e81-683c-4c55-b8a8-c8ccbdab854d None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:38.961841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0921e81-683c-4c55-b8a8-c8ccbdab854d None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:38.962148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0921e81-683c-4c55-b8a8-c8ccbdab854d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:38.984885 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c0921e81-683c-4c55-b8a8-c8ccbdab854d None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:38.986156 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c0921e81-683c-4c55-b8a8-c8ccbdab854d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:38.995179 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1294/2613] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:39.000172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a72cf8c-6e41-4181-8be7-e2b982087762 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.010677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a72cf8c-6e41-4181-8be7-e2b982087762 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.016072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.016690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.016971 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.017281 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.017945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:34:39.018246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.019858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.033081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1320/2614] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:34:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:34:39.037868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0921e81-683c-4c55-b8a8-c8ccbdab854d None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.050992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0921e81-683c-4c55-b8a8-c8ccbdab854d None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.057363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.057989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.058402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.058537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.059358 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.061826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.097716 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1295/2615] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:39 2022] POST /identity/v3/projects => generated 373 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:39.103406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.115541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b728a-0430-4d67-8fd8-423db4808a90 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.175714 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:39.194203 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.194813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.195123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.195409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.196046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.198747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.223659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1321/2616] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:39 2022] POST /identity/v3/users => generated 287 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:39.228657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.240034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2323f30-5a4a-449e-b1fb-6fe2732a7466 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.245957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.246586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.246906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.247114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.247789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.250122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.261080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1296/2617] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:34:39.265919 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.276438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d138d9-0ff1-40e0-b530-8c1d6f930313 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.281643 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a011e86-bec1-44f7-8c4b-b48b0b124736 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.282254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a011e86-bec1-44f7-8c4b-b48b0b124736 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.282522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a011e86-bec1-44f7-8c4b-b48b0b124736 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.282784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a011e86-bec1-44f7-8c4b-b48b0b124736 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.283503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a011e86-bec1-44f7-8c4b-b48b0b124736 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.285219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a011e86-bec1-44f7-8c4b-b48b0b124736 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.297912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1322/2618] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:39.302186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.313772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4405e2c5-09dc-4085-984b-9fe8c506e903 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.319279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.320090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.320222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.320525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] PATH_INFO: `/v3/projects/5d046a832eac49fda2b3efd3315cda88/users/03f2912b300d4db392def0d602019c65/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.338365 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d046a832eac49fda2b3efd3315cda88, user_id=03f2912b300d4db392def0d602019c65, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.341090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.358741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1297/2619] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:39 2022] PUT /identity/v3/projects/5d046a832eac49fda2b3efd3315cda88/users/03f2912b300d4db392def0d602019c65/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:34:39.370096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422773b6-60b6-4633-b16b-067e47a828ce None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.370734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422773b6-60b6-4633-b16b-067e47a828ce None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.370734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422773b6-60b6-4633-b16b-067e47a828ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.403047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-422773b6-60b6-4633-b16b-067e47a828ce None None] MFA Rules not processed for user `03f2912b300d4db392def0d602019c65`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:39.404352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-422773b6-60b6-4633-b16b-067e47a828ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:39.477654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1323/2620] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:39.492108 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.493942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:39.575681 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be42ddc-74d5-4816-8253-6b3539b01726 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.581474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7589a2-1d39-4660-b900-4b08f6a38f22 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.582075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7589a2-1d39-4660-b900-4b08f6a38f22 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.582342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7589a2-1d39-4660-b900-4b08f6a38f22 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.582602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7589a2-1d39-4660-b900-4b08f6a38f22 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.590975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7589a2-1d39-4660-b900-4b08f6a38f22 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.592616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7589a2-1d39-4660-b900-4b08f6a38f22 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.602399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1298/2621] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:39 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:39.849047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422773b6-60b6-4633-b16b-067e47a828ce None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:39.922861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422773b6-60b6-4633-b16b-067e47a828ce None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:39.927500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:39.928135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:39.928409 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:39.928681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] PATH_INFO: `/v3/users/03f2912b300d4db392def0d602019c65` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:39.929828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03f2912b300d4db392def0d602019c65)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:39.931746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:39.990717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03f2912b300d4db392def0d602019c65'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:39.999823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:40.005038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03f2912b300d4db392def0d602019c65'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.005886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03f2912b300d4db392def0d602019c65'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.015059 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03f2912b300d4db392def0d602019c65'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.025596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03f2912b300d4db392def0d602019c65'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.033241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1324/2622] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:34:39 2022] DELETE /identity/v3/users/03f2912b300d4db392def0d602019c65 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:40.039144 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:40.039742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:40.039742 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:40.062727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:40.064073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:40.129659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:40.146890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1299/2623] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:40.158111 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:40.159594 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:40.230333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae90024-b4af-4a7f-892f-112f57946588 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:40.234636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e08c63f-fcd4-4a81-b3c4-57ba67c65dee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:40.235296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e08c63f-fcd4-4a81-b3c4-57ba67c65dee service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:40.235592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e08c63f-fcd4-4a81-b3c4-57ba67c65dee service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:40.235858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e08c63f-fcd4-4a81-b3c4-57ba67c65dee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:40.250148 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e08c63f-fcd4-4a81-b3c4-57ba67c65dee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:40.251524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e08c63f-fcd4-4a81-b3c4-57ba67c65dee service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:40.261823 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1325/2624] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:34:40.568445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:40.570319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:40.584306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d5bfc1-e5eb-4048-948b-4bc0f0c1342b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:40.591176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:40.591976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:40.592264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:40.592557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] PATH_INFO: `/v3/projects/5d046a832eac49fda2b3efd3315cda88` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:40.594124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d046a832eac49fda2b3efd3315cda88)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:40.596553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:40.657542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d046a832eac49fda2b3efd3315cda88', 'request_id': 'req-f48556d2-f21a-4e44-9a5b-548e912fc5e0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.666163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d046a832eac49fda2b3efd3315cda88', 'request_id': 'req-f48556d2-f21a-4e44-9a5b-548e912fc5e0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.673819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d046a832eac49fda2b3efd3315cda88', 'request_id': 'req-f48556d2-f21a-4e44-9a5b-548e912fc5e0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.684644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] The token cache is being invalidate because project 5d046a832eac49fda2b3efd3315cda88 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:34:40.684949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.685685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-f48556d2-f21a-4e44-9a5b-548e912fc5e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:40.687207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1300/2625] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:34:40 2022] DELETE /identity/v3/projects/5d046a832eac49fda2b3efd3315cda88 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:40.692222 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59b484b-338e-4b77-b42c-0f99145eb831 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:40.692889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59b484b-338e-4b77-b42c-0f99145eb831 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:40.692889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59b484b-338e-4b77-b42c-0f99145eb831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:40.713038 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c59b484b-338e-4b77-b42c-0f99145eb831 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:40.714327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c59b484b-338e-4b77-b42c-0f99145eb831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:40.799044 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1326/2626] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:40.807666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30a61c-2541-4b92-8ee1-ff8df9199429 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:40.807997 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30a61c-2541-4b92-8ee1-ff8df9199429 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:40.808284 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30a61c-2541-4b92-8ee1-ff8df9199429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:40.829412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5f30a61c-2541-4b92-8ee1-ff8df9199429 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:40.830618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5f30a61c-2541-4b92-8ee1-ff8df9199429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:40.840942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1301/2627] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:40.845632 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59b484b-338e-4b77-b42c-0f99145eb831 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:40.859552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59b484b-338e-4b77-b42c-0f99145eb831 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:40.864485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:40.865249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:40.865624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:40.865959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:40.866675 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:34:40.867024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:40.868746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:40.881832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1327/2628] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:34:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:34:40.886559 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f30a61c-2541-4b92-8ee1-ff8df9199429 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:40.896722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f30a61c-2541-4b92-8ee1-ff8df9199429 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:40.902255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:40.902863 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:40.903136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:40.903465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:40.904205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:40.906172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:40.941199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1302/2629] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:40 2022] POST /identity/v3/projects => generated 365 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:40.946233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:40.963611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443e4b31-d0be-42b3-948e-b220c28eb363 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.034279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.034860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.035149 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.035837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.036485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.038429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.050461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:41.065858 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1328/2630] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:40 2022] POST /identity/v3/users => generated 283 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:41.070645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.082620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa2e62-73e1-4760-8751-09671d5e5596 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.087726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.088352 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.088623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.088895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.089589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.091196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.102688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1303/2631] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:41.108849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.121662 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874ba480-47bc-4c56-8dba-c4610371d0de admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.128479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9583f530-d97c-495b-ab55-ab92c25d7cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.129108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9583f530-d97c-495b-ab55-ab92c25d7cbe admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.129408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9583f530-d97c-495b-ab55-ab92c25d7cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.129674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9583f530-d97c-495b-ab55-ab92c25d7cbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.130325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9583f530-d97c-495b-ab55-ab92c25d7cbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.132010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9583f530-d97c-495b-ab55-ab92c25d7cbe admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.144345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1329/2632] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:41 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:41.148838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.160063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8186014-fefd-4c85-961d-c8c98eaf8ef0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.165602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.166333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.166605 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.166877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] PATH_INFO: `/v3/projects/b41b4359c96a483086ae4bfeca57b822/users/3e1156eaae1b483b9bbe1f79ae8e122a/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.182753 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b41b4359c96a483086ae4bfeca57b822, user_id=3e1156eaae1b483b9bbe1f79ae8e122a, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.185474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.201198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1304/2633] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:41 2022] PUT /identity/v3/projects/b41b4359c96a483086ae4bfeca57b822/users/3e1156eaae1b483b9bbe1f79ae8e122a/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:41.212306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb56e5e-006e-460d-9459-abcc7f768c3d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.212588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb56e5e-006e-460d-9459-abcc7f768c3d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.212892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb56e5e-006e-460d-9459-abcc7f768c3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.244270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-9eb56e5e-006e-460d-9459-abcc7f768c3d None None] MFA Rules not processed for user `3e1156eaae1b483b9bbe1f79ae8e122a`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:41.245529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9eb56e5e-006e-460d-9459-abcc7f768c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:41.313593 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1330/2634] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:41 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:41.318213 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.391715 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f796fa62-0ffb-4e00-a2cf-17fee26bda22 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.396561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.397185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.397486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.397752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.398410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.400308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.424537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:41.432670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1305/2635] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:41 2022] POST /identity/v3/projects => generated 363 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:41.439624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb56e5e-006e-460d-9459-abcc7f768c3d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.452966 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb56e5e-006e-460d-9459-abcc7f768c3d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.530792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.531429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.531697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.531963 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.532626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.534585 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.558284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1331/2636] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:41 2022] POST /identity/v3/users => generated 288 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:34:41.563498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.574810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc7453f-a211-4ace-8970-a247fd4415b0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.580193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.580829 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.581104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.581477 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.582172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.583805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.595632 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1306/2637] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:34:41.601949 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.612821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13366fd5-2467-454b-87c8-05fbe6af8589 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.618468 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.619189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.619512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.619827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] PATH_INFO: `/v3/projects/a701e0c9db8341a3ba82d43b1d5d3782/users/65bada6af67b4856b96b363200f87fe1/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.643658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a701e0c9db8341a3ba82d43b1d5d3782, user_id=65bada6af67b4856b96b363200f87fe1, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.646336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.661323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1332/2638] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:41 2022] PUT /identity/v3/projects/a701e0c9db8341a3ba82d43b1d5d3782/users/65bada6af67b4856b96b363200f87fe1/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:41.666377 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.677056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee0150-8c0e-46ca-9fdb-a723786b1166 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.736736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c967e21-cbc5-4472-824c-8cc337f71c33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.737411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c967e21-cbc5-4472-824c-8cc337f71c33 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.737659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c967e21-cbc5-4472-824c-8cc337f71c33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.737921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c967e21-cbc5-4472-824c-8cc337f71c33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.738579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c967e21-cbc5-4472-824c-8cc337f71c33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.740195 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c967e21-cbc5-4472-824c-8cc337f71c33 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.751684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1307/2639] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:41 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:41.756185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.769381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f4479-4029-4d03-8dd2-249d5eae5ae2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:41.775024 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:41.775752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.776035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.776298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] PATH_INFO: `/v3/projects/a701e0c9db8341a3ba82d43b1d5d3782/users/65bada6af67b4856b96b363200f87fe1/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.778540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=a701e0c9db8341a3ba82d43b1d5d3782, user_id=65bada6af67b4856b96b363200f87fe1, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:41.781656 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:41.793466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1333/2640] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:41 2022] PUT /identity/v3/projects/a701e0c9db8341a3ba82d43b1d5d3782/users/65bada6af67b4856b96b363200f87fe1/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:41.805665 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727be299-e934-4b9c-b386-99a83a00d5ac None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:41.805999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727be299-e934-4b9c-b386-99a83a00d5ac None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:41.806255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727be299-e934-4b9c-b386-99a83a00d5ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:41.846511 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-727be299-e934-4b9c-b386-99a83a00d5ac None None] MFA Rules not processed for user `65bada6af67b4856b96b363200f87fe1`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:41.847593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-727be299-e934-4b9c-b386-99a83a00d5ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:41.921316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1308/2641] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:41 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:41.937416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:41.939197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:41.977717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:42.007127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73f253b-29a8-4c67-a228-caec9af635ae admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:42.011034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:42.011639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:42.011894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:42.012157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:42.021574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:42.022999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:42.032471 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1334/2642] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:41 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:34:42.117924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727be299-e934-4b9c-b386-99a83a00d5ac None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:42.130388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727be299-e934-4b9c-b386-99a83a00d5ac None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:42.133134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d87ba7b-6024-4f56-be3a-cfac2aaa5b37 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:42.133719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87ba7b-6024-4f56-be3a-cfac2aaa5b37 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:42.133987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87ba7b-6024-4f56-be3a-cfac2aaa5b37 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:42.134244 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87ba7b-6024-4f56-be3a-cfac2aaa5b37 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:42.192974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d87ba7b-6024-4f56-be3a-cfac2aaa5b37 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:42.194448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d87ba7b-6024-4f56-be3a-cfac2aaa5b37 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:42.203580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1309/2643] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:42 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:45.019665 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:45.088219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38a4e6-211e-44a7-a362-1ebabfecb72c service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:45.094233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:45.094862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:45.095132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:45.095397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] PATH_INFO: `/v3/users/3e1156eaae1b483b9bbe1f79ae8e122a` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:45.096630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e1156eaae1b483b9bbe1f79ae8e122a)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:45.098540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:45.147163 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e1156eaae1b483b9bbe1f79ae8e122a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.160433 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e1156eaae1b483b9bbe1f79ae8e122a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.161282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e1156eaae1b483b9bbe1f79ae8e122a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.170546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e1156eaae1b483b9bbe1f79ae8e122a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.178707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e1156eaae1b483b9bbe1f79ae8e122a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.185568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1335/2644] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:34:45 2022] DELETE /identity/v3/users/3e1156eaae1b483b9bbe1f79ae8e122a => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:45.190439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:45.191005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:45.191005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:45.208848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=100 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:45.215867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:45.217077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:45.283871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1310/2645] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:45.295412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:45.297110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:45.353282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:45.359041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522d66cd-90db-4fbd-841a-6c7a8b67e201 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:45.366373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:45.367314 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:45.367584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:45.367874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:45.378248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:45.379678 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:45.389459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1336/2646] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:45.674879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:45.676401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:45.689283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7210516-387f-493d-9936-15fc2c1ac656 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:45.693895 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:45.694597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:45.694875 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:45.695142 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] PATH_INFO: `/v3/projects/b41b4359c96a483086ae4bfeca57b822` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:45.696285 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b41b4359c96a483086ae4bfeca57b822)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:45.698247 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:45.750136 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b41b4359c96a483086ae4bfeca57b822', 'request_id': 'req-df5eec59-3f9c-4257-8145-ec214395d785'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.759010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b41b4359c96a483086ae4bfeca57b822', 'request_id': 'req-df5eec59-3f9c-4257-8145-ec214395d785'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.766027 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b41b4359c96a483086ae4bfeca57b822', 'request_id': 'req-df5eec59-3f9c-4257-8145-ec214395d785'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.776496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] The token cache is being invalidate because project b41b4359c96a483086ae4bfeca57b822 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:34:45.776760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.777439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.778970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1311/2647] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:34:45 2022] DELETE /identity/v3/projects/b41b4359c96a483086ae4bfeca57b822 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:45.783101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:45.792899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ea85d2-a8e2-4ac7-9072-f665bc3623d4 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:45.865712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:45.866360 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:45.866639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:45.866901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] PATH_INFO: `/v3/users/65bada6af67b4856b96b363200f87fe1` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:45.868132 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=65bada6af67b4856b96b363200f87fe1)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:45.870258 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:45.930558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65bada6af67b4856b96b363200f87fe1'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.942284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65bada6af67b4856b96b363200f87fe1'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.942958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65bada6af67b4856b96b363200f87fe1'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.953588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65bada6af67b4856b96b363200f87fe1'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.960456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-341ab07f-d978-442e-b68d-71d2c5790b6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65bada6af67b4856b96b363200f87fe1'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:45.966507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1337/2648] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:34:45 2022] DELETE /identity/v3/users/65bada6af67b4856b96b363200f87fe1 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:46.260644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:46.262798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:46.283049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5eec59-3f9c-4257-8145-ec214395d785 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:46.342500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:46.343190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.343462 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.343726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] PATH_INFO: `/v3/projects/a701e0c9db8341a3ba82d43b1d5d3782` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.345009 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a701e0c9db8341a3ba82d43b1d5d3782)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:46.346973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:46.382354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:46.396577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a701e0c9db8341a3ba82d43b1d5d3782', 'request_id': 'req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:46.405330 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a701e0c9db8341a3ba82d43b1d5d3782', 'request_id': 'req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:46.411707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a701e0c9db8341a3ba82d43b1d5d3782', 'request_id': 'req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:46.423154 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] The token cache is being invalidate because project a701e0c9db8341a3ba82d43b1d5d3782 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:34:46.423439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:46.424149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-90d44f2d-ec9f-41b8-92b2-9d73dd8a12a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:46.425757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1312/2649] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:34:46 2022] DELETE /identity/v3/projects/a701e0c9db8341a3ba82d43b1d5d3782 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:46.430851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5725842f-0204-4b4c-afc2-5f74efaf7c3e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.431348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5725842f-0204-4b4c-afc2-5f74efaf7c3e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.431458 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5725842f-0204-4b4c-afc2-5f74efaf7c3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.453522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5725842f-0204-4b4c-afc2-5f74efaf7c3e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:46.454681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5725842f-0204-4b4c-afc2-5f74efaf7c3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:46.519553 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1338/2650] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:46.530584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65441ab3-0174-4e9c-ad54-3ac809cd3a90 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.530868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65441ab3-0174-4e9c-ad54-3ac809cd3a90 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.531139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65441ab3-0174-4e9c-ad54-3ac809cd3a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.551988 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-65441ab3-0174-4e9c-ad54-3ac809cd3a90 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:46.553234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-65441ab3-0174-4e9c-ad54-3ac809cd3a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:46.562328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1313/2651] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:46.566783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5725842f-0204-4b4c-afc2-5f74efaf7c3e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:46.576937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5725842f-0204-4b4c-afc2-5f74efaf7c3e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:46.582186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:46.582781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.583053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.583321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.583954 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:34:46.584249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:46.585891 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:46.592711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:46.598969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1339/2652] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:34:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:34:46.603368 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65441ab3-0174-4e9c-ad54-3ac809cd3a90 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:46.613175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65441ab3-0174-4e9c-ad54-3ac809cd3a90 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:46.618592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:46.619194 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.619468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.619731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.620371 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:46.622294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:46.652566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1314/2653] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:46 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:46.657613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:46.675172 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792831b9-bcb6-4c40-bc71-0c569802f49c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:46.743921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:46.744681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.745325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.745325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.745900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:46.747829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:46.774784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1340/2654] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:46 2022] POST /identity/v3/users => generated 293 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:46.776538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:46.787091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed401c-38f4-49df-8645-0779e753fd29 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:46.791737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:46.792331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.792601 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.792862 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.793873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:46.795219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:46.806479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1315/2655] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:46.810959 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:46.826255 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d1830c-97bc-4ca7-af4a-aef85c788ad4 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:46.832243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3953a27a-c6bb-4a02-b5dc-9706e7a6825f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:46.832841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3953a27a-c6bb-4a02-b5dc-9706e7a6825f admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.833114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3953a27a-c6bb-4a02-b5dc-9706e7a6825f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.833415 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3953a27a-c6bb-4a02-b5dc-9706e7a6825f admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.834058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3953a27a-c6bb-4a02-b5dc-9706e7a6825f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:46.835653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3953a27a-c6bb-4a02-b5dc-9706e7a6825f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:46.847864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1341/2656] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:46 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:46.855480 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:46.870867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ed931e-c801-44d3-804c-642b60321884 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:46.877611 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:46.878215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.878488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.878762 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] PATH_INFO: `/v3/projects/8f56696af42c473db9b7c0b4208c3a29/users/e39589f98bdb44b8bcbfb25fb30a2159/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.895971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f56696af42c473db9b7c0b4208c3a29, user_id=e39589f98bdb44b8bcbfb25fb30a2159, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:46.898670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:46.919738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1316/2657] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:46 2022] PUT /identity/v3/projects/8f56696af42c473db9b7c0b4208c3a29/users/e39589f98bdb44b8bcbfb25fb30a2159/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:46.935034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1440b39-c646-40c6-8d86-9ae4d724e68c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:46.935336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1440b39-c646-40c6-8d86-9ae4d724e68c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:46.935605 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1440b39-c646-40c6-8d86-9ae4d724e68c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:46.965626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c1440b39-c646-40c6-8d86-9ae4d724e68c None None] MFA Rules not processed for user `e39589f98bdb44b8bcbfb25fb30a2159`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:46.966836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c1440b39-c646-40c6-8d86-9ae4d724e68c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:47.046785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1342/2658] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:47.051532 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.122028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477e8ba1-3c19-425a-9630-003f6f803eb5 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.127300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.127901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.128173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.128437 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.129066 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.130975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.164873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1317/2659] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:47 2022] POST /identity/v3/projects => generated 383 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:34:47.169720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1440b39-c646-40c6-8d86-9ae4d724e68c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.186663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1440b39-c646-40c6-8d86-9ae4d724e68c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.256877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.257559 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.257825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.258092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.258772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.260772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.272442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:47.286319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1343/2660] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:47 2022] POST /identity/v3/users => generated 292 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:47.292225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.303341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0da06-aece-4e23-865d-294d8c0ee0e7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.308850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.309489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.309779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.310042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.310687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.312836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.324888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1318/2661] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:47.329362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.339839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49861391-e1a8-4e2b-9f30-59bc8a27a244 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.345912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3622cff-0293-4c0b-b6b9-55251c2dba8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.346516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3622cff-0293-4c0b-b6b9-55251c2dba8c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.346792 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3622cff-0293-4c0b-b6b9-55251c2dba8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.347061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3622cff-0293-4c0b-b6b9-55251c2dba8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.347704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3622cff-0293-4c0b-b6b9-55251c2dba8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.349327 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3622cff-0293-4c0b-b6b9-55251c2dba8c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.361300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1344/2662] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:47.367545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.382757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4f34e-42d5-434e-a9af-f1eba3594ea9 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.390576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.391250 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.391522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.391811 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] PATH_INFO: `/v3/projects/bb26164aeefc4b9ba6de5b6ca4101d04/users/41e0c3d1c04246798e8324515db08f33/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.408506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb26164aeefc4b9ba6de5b6ca4101d04, user_id=41e0c3d1c04246798e8324515db08f33, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.411161 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.426329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1319/2663] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:47 2022] PUT /identity/v3/projects/bb26164aeefc4b9ba6de5b6ca4101d04/users/41e0c3d1c04246798e8324515db08f33/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:47.438431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72810794-2648-4007-b4e8-434d57e5d4e1 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.438931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72810794-2648-4007-b4e8-434d57e5d4e1 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.438931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72810794-2648-4007-b4e8-434d57e5d4e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.470840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-72810794-2648-4007-b4e8-434d57e5d4e1 None None] MFA Rules not processed for user `41e0c3d1c04246798e8324515db08f33`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:47.472681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-72810794-2648-4007-b4e8-434d57e5d4e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:47.546796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1345/2664] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:47 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:47.553526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.566208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:47.622752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c91d97-72ec-47a4-9f19-3f4bc20316bc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.628557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.629204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.629474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.629735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.630367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.632249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.665941 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1320/2665] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:47 2022] POST /identity/v3/projects => generated 385 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:47.670127 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72810794-2648-4007-b4e8-434d57e5d4e1 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.681497 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72810794-2648-4007-b4e8-434d57e5d4e1 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.749592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.750511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.750856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.751236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.752238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.755410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.784103 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1346/2666] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:47 2022] POST /identity/v3/users => generated 299 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:34:47.788375 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.799124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08065ae-c07b-417a-9766-1747a94fd031 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.803994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.804531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.804801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.805062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.805751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.807327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.818442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1321/2667] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:47.822765 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.833374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19496244-2522-4104-ba76-b943c28c1174 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.839041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.839698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.839974 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.840247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] PATH_INFO: `/v3/projects/64bb73504dc84eb486568bd1aefd545f/users/c99e7d188f4c48cab041d02da2df76bf/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.859393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64bb73504dc84eb486568bd1aefd545f, user_id=c99e7d188f4c48cab041d02da2df76bf, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.863674 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.888842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1347/2668] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:47 2022] PUT /identity/v3/projects/64bb73504dc84eb486568bd1aefd545f/users/c99e7d188f4c48cab041d02da2df76bf/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:47.894726 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:47.910595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864d5d8-f260-4ba8-ae7a-17b93d490a5e admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:47.984413 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4fcc8-8e8a-4153-a452-6de27047ee09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:47.985026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4fcc8-8e8a-4153-a452-6de27047ee09 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:47.985465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4fcc8-8e8a-4153-a452-6de27047ee09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:47.985735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4fcc8-8e8a-4153-a452-6de27047ee09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:47.986387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4fcc8-8e8a-4153-a452-6de27047ee09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:47.987973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4fcc8-8e8a-4153-a452-6de27047ee09 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:47.998999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1322/2669] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:47 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:48.003710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:48.014806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edbce43-ec5f-48c6-954b-7273ecd18040 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:48.020367 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:48.021014 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:48.021311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:48.021590 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] PATH_INFO: `/v3/projects/64bb73504dc84eb486568bd1aefd545f/users/c99e7d188f4c48cab041d02da2df76bf/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:48.023773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=64bb73504dc84eb486568bd1aefd545f, user_id=c99e7d188f4c48cab041d02da2df76bf, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:48.026420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:48.037681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1348/2670] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:48 2022] PUT /identity/v3/projects/64bb73504dc84eb486568bd1aefd545f/users/c99e7d188f4c48cab041d02da2df76bf/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:48.050461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:48.050841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:48.051192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:48.103797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] MFA Rules not processed for user `c99e7d188f4c48cab041d02da2df76bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:48.105355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:48.189798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1323/2671] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:48 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:34:48.200911 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:48.202912 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:48.236561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:48.267927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213eeed9-4ac4-480f-8318-0d694e7915ac admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:48.271782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:48.272400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:48.272695 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:48.272944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:48.282802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:48.284286 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:48.294306 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1349/2672] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:48 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:48.887814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:48.889530 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:48.899131 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:48.909754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6581d9-8b82-4f51-8442-f7f0d48245f3 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:48.919998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:48.921385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:48.921848 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:48.922275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:48.998749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:49.003775 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:49.012636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1324/2673] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:48 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:53.213888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:53.223348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e646b9aa-46e6-41a4-bc50-62ee10033ad6 service cinder] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:53.227484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e0f3ba-6199-4d55-9c85-880acc512890 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:53.228140 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e0f3ba-6199-4d55-9c85-880acc512890 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:53.228414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e0f3ba-6199-4d55-9c85-880acc512890 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:53.228684 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e0f3ba-6199-4d55-9c85-880acc512890 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:53.302969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e0f3ba-6199-4d55-9c85-880acc512890 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:53.304418 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e0f3ba-6199-4d55-9c85-880acc512890 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:53.318354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1350/2674] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:53 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:55.707990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:55.773830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804987b1-e335-45ea-a4e1-4d2780743358 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:55.781407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:55.782055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:55.782327 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:55.782590 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] PATH_INFO: `/v3/users/e39589f98bdb44b8bcbfb25fb30a2159` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:55.783740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e39589f98bdb44b8bcbfb25fb30a2159)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:55.785695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:55.850159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e39589f98bdb44b8bcbfb25fb30a2159'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:55.858090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e39589f98bdb44b8bcbfb25fb30a2159'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:55.865903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e39589f98bdb44b8bcbfb25fb30a2159'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:55.866534 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e39589f98bdb44b8bcbfb25fb30a2159'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:55.867138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e39589f98bdb44b8bcbfb25fb30a2159'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:55.886065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1325/2675] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:34:55 2022] DELETE /identity/v3/users/e39589f98bdb44b8bcbfb25fb30a2159 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:55.891043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:55.891540 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:55.891654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:55.912466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:55.913770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:55.985626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1351/2676] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:55.996584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:55.998201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:56.021282 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:56.064624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9e39fa-9c2b-423f-8477-3ac94b1296d0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:56.068378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:56.068989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:56.069298 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:56.069567 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:56.079374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:56.080799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:56.090710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1326/2677] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:34:56.387337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:56.388894 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:56.404025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ea23c-427a-4f45-adf0-8c0388778cf6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:56.410121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:56.410756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:56.411357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:56.411629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] PATH_INFO: `/v3/projects/8f56696af42c473db9b7c0b4208c3a29` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:56.412842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f56696af42c473db9b7c0b4208c3a29)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:56.414769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:56.469407 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f56696af42c473db9b7c0b4208c3a29', 'request_id': 'req-64fa7375-619e-4211-86cb-e0a514000f92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.477312 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f56696af42c473db9b7c0b4208c3a29', 'request_id': 'req-64fa7375-619e-4211-86cb-e0a514000f92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.483642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f56696af42c473db9b7c0b4208c3a29', 'request_id': 'req-64fa7375-619e-4211-86cb-e0a514000f92'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.485690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:56.505330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] The token cache is being invalidate because project 8f56696af42c473db9b7c0b4208c3a29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:34:56.505739 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.506639 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.508473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1352/2678] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:34:56 2022] DELETE /identity/v3/projects/8f56696af42c473db9b7c0b4208c3a29 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:56.512322 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:56.526204 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9814fa8a-b7f1-4d45-9ac1-7b7d2674b7fd service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:56.598121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:56.598760 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:56.599047 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:56.599303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] PATH_INFO: `/v3/users/41e0c3d1c04246798e8324515db08f33` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:56.600613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41e0c3d1c04246798e8324515db08f33)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:56.602899 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:56.652701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41e0c3d1c04246798e8324515db08f33'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.660926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41e0c3d1c04246798e8324515db08f33'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.671624 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41e0c3d1c04246798e8324515db08f33'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.672389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41e0c3d1c04246798e8324515db08f33'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.673008 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41e0c3d1c04246798e8324515db08f33'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:56.692981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1327/2679] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:34:56 2022] DELETE /identity/v3/users/41e0c3d1c04246798e8324515db08f33 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:56.992200 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:56.993806 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:57.009485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa7375-619e-4211-86cb-e0a514000f92 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:57.086445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:57.087079 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:57.087354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:57.087625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] PATH_INFO: `/v3/projects/bb26164aeefc4b9ba6de5b6ca4101d04` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:57.088793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb26164aeefc4b9ba6de5b6ca4101d04)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:57.090741 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:57.139953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb26164aeefc4b9ba6de5b6ca4101d04', 'request_id': 'req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.146984 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb26164aeefc4b9ba6de5b6ca4101d04', 'request_id': 'req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.152542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb26164aeefc4b9ba6de5b6ca4101d04', 'request_id': 'req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.166157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] The token cache is being invalidate because project bb26164aeefc4b9ba6de5b6ca4101d04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:34:57.166442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.167161 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.168905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1353/2680] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:34:56 2022] DELETE /identity/v3/projects/bb26164aeefc4b9ba6de5b6ca4101d04 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:57.173273 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:57.174729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:57.188060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb731da-b6a7-4eaa-aa09-fab470b19248 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:57.232830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:57.251859 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:57.252525 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:57.252799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:57.253067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] PATH_INFO: `/v3/users/c99e7d188f4c48cab041d02da2df76bf` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:57.254281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c99e7d188f4c48cab041d02da2df76bf)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:57.256231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:57.313754 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c99e7d188f4c48cab041d02da2df76bf'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.322127 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c99e7d188f4c48cab041d02da2df76bf'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.331191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c99e7d188f4c48cab041d02da2df76bf'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.331806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c99e7d188f4c48cab041d02da2df76bf'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.332433 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-e209648c-1421-406d-bb81-298b6a6d841e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c99e7d188f4c48cab041d02da2df76bf'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.350677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1328/2681] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:34:57 2022] DELETE /identity/v3/users/c99e7d188f4c48cab041d02da2df76bf => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:57.653769 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:57.655392 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:57.672826 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b87a30-58a3-4dfa-b0c8-b6c7dca715ec admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:57.735492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:57.736137 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:57.736413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:57.736682 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] PATH_INFO: `/v3/projects/64bb73504dc84eb486568bd1aefd545f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:57.737846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=64bb73504dc84eb486568bd1aefd545f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:57.739774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:57.765970 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:INSERT=2,keystone:DELETE=9,keystone:SELECT=89 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:57.793217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64bb73504dc84eb486568bd1aefd545f', 'request_id': 'req-bde00b6e-6d29-4d83-99a4-c8da749226bb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.800786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64bb73504dc84eb486568bd1aefd545f', 'request_id': 'req-bde00b6e-6d29-4d83-99a4-c8da749226bb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.806247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64bb73504dc84eb486568bd1aefd545f', 'request_id': 'req-bde00b6e-6d29-4d83-99a4-c8da749226bb'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.820839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] The token cache is being invalidate because project 64bb73504dc84eb486568bd1aefd545f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:34:57.821248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.822030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bde00b6e-6d29-4d83-99a4-c8da749226bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:34:57.823813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1354/2682] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:34:57 2022] DELETE /identity/v3/projects/64bb73504dc84eb486568bd1aefd545f => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:57.830827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c217dd00-ce2b-4075-b8f2-0092e36fbdf7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:57.831112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c217dd00-ce2b-4075-b8f2-0092e36fbdf7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:57.831384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c217dd00-ce2b-4075-b8f2-0092e36fbdf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:57.855677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c217dd00-ce2b-4075-b8f2-0092e36fbdf7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:57.857646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c217dd00-ce2b-4075-b8f2-0092e36fbdf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:57.933001 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1329/2683] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:57.943006 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d950a701-0697-4d34-9003-8528a43c001f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:57.943452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d950a701-0697-4d34-9003-8528a43c001f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:57.943730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d950a701-0697-4d34-9003-8528a43c001f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:57.964009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d950a701-0697-4d34-9003-8528a43c001f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:57.965320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d950a701-0697-4d34-9003-8528a43c001f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:57.973732 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1355/2684] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:57.978748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c217dd00-ce2b-4075-b8f2-0092e36fbdf7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:57.988619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c217dd00-ce2b-4075-b8f2-0092e36fbdf7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:57.993251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:57.993841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:57.994109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:57.994373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:57.995166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:34:57.995890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:57.997019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.012537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1330/2685] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:34:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:34:58.017731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d950a701-0697-4d34-9003-8528a43c001f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.029608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d950a701-0697-4d34-9003-8528a43c001f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.035655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.036388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.036668 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.036931 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.037599 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.039740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.072376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1356/2686] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:58 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:58.076688 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.087424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51a85d1-8482-428c-b2e7-8cdc096d443a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.141037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:58.158056 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.158920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.159308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.159697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.160699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.163877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.190061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1331/2687] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:58 2022] POST /identity/v3/users => generated 292 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:58.194338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.205922 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c32c589-4fba-4540-a773-dc8dc0120a74 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.210555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65544016-0975-483c-9173-3230b979bdda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.211150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65544016-0975-483c-9173-3230b979bdda admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.211420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65544016-0975-483c-9173-3230b979bdda admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.211681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65544016-0975-483c-9173-3230b979bdda admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.212329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65544016-0975-483c-9173-3230b979bdda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.213995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65544016-0975-483c-9173-3230b979bdda admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.229488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1357/2688] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:58 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:58.234129 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.245353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544e81fc-2aad-4116-a753-c8abb1971de5 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.251121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34ab8e4-eb40-49ae-8e7d-1cce7c9f2dfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.251727 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ab8e4-eb40-49ae-8e7d-1cce7c9f2dfb admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.251998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ab8e4-eb40-49ae-8e7d-1cce7c9f2dfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.252263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ab8e4-eb40-49ae-8e7d-1cce7c9f2dfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.252921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34ab8e4-eb40-49ae-8e7d-1cce7c9f2dfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.254930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34ab8e4-eb40-49ae-8e7d-1cce7c9f2dfb admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.267037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1332/2689] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:58.271555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65544016-0975-483c-9173-3230b979bdda admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.283141 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65544016-0975-483c-9173-3230b979bdda admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.289184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.289889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.290160 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.290428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] PATH_INFO: `/v3/projects/bad6527e51174df28f58ce373e0d9651/users/c9726973c66448eeb588aa85e9e18b4a/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.311113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bad6527e51174df28f58ce373e0d9651, user_id=c9726973c66448eeb588aa85e9e18b4a, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.314273 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.328595 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1358/2690] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:58 2022] PUT /identity/v3/projects/bad6527e51174df28f58ce373e0d9651/users/c9726973c66448eeb588aa85e9e18b4a/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:58.341838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aaf8a5-3619-41b7-9024-cf0a4d6f09db None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.342133 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aaf8a5-3619-41b7-9024-cf0a4d6f09db None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.342386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aaf8a5-3619-41b7-9024-cf0a4d6f09db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.378746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-70aaf8a5-3619-41b7-9024-cf0a4d6f09db None None] MFA Rules not processed for user `c9726973c66448eeb588aa85e9e18b4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:58.380059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-70aaf8a5-3619-41b7-9024-cf0a4d6f09db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:58.465474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1333/2691] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:58.470565 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.553429 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d8b4ab-295b-4d69-8a9e-0a85fc4edd1c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.559292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.559927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.560233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.560554 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.561298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.564139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.594445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1359/2692] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:34:58 2022] POST /identity/v3/projects => generated 383 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:58.598975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70aaf8a5-3619-41b7-9024-cf0a4d6f09db None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.611610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70aaf8a5-3619-41b7-9024-cf0a4d6f09db None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.673132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.673763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.674035 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.674302 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.674973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.676910 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.700248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1334/2693] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:34:58 2022] POST /identity/v3/users => generated 298 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:34:58.706620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.720190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a6db54-c2e8-48a9-bdb5-cf32ce520ac5 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.725896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.726522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.726793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.727396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.728054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.729698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.741561 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1360/2694] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:58 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:58.746160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.757168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30296ca0-35a9-4718-82ba-1d9660fe9f18 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.771306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.772296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.772631 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.772962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] PATH_INFO: `/v3/projects/76971985b5aa41f2aabf1a7a1a37dd13/users/75bdf07ecad140c6aff3ae32a08e37ec/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.793055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76971985b5aa41f2aabf1a7a1a37dd13, user_id=75bdf07ecad140c6aff3ae32a08e37ec, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.795806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.812500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1335/2695] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:58 2022] PUT /identity/v3/projects/76971985b5aa41f2aabf1a7a1a37dd13/users/75bdf07ecad140c6aff3ae32a08e37ec/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:58.817214 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.827802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3a2280-6be9-416b-9293-ada8e4f9a97b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.850776 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:58.897873 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2fd1ab-800e-462f-9175-b5a117a25521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.898492 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fd1ab-800e-462f-9175-b5a117a25521 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.898766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fd1ab-800e-462f-9175-b5a117a25521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.899036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fd1ab-800e-462f-9175-b5a117a25521 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.899707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fd1ab-800e-462f-9175-b5a117a25521 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.901388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fd1ab-800e-462f-9175-b5a117a25521 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.912940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1361/2696] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:34:58 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:34:58.917884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:58.928697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46211d3e-ea9d-458d-af49-8a36ec539458 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:58.933950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:58.934586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.934856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.935124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] PATH_INFO: `/v3/projects/76971985b5aa41f2aabf1a7a1a37dd13/users/75bdf07ecad140c6aff3ae32a08e37ec/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.937091 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76971985b5aa41f2aabf1a7a1a37dd13, user_id=75bdf07ecad140c6aff3ae32a08e37ec, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:58.939853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:58.951125 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1336/2697] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:34:58 2022] PUT /identity/v3/projects/76971985b5aa41f2aabf1a7a1a37dd13/users/75bdf07ecad140c6aff3ae32a08e37ec/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:34:58.961743 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4093e723-0f85-4dd5-b6ad-76608f084dea None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:58.962052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4093e723-0f85-4dd5-b6ad-76608f084dea None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:58.962364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4093e723-0f85-4dd5-b6ad-76608f084dea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:58.995627 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-4093e723-0f85-4dd5-b6ad-76608f084dea None None] MFA Rules not processed for user `75bdf07ecad140c6aff3ae32a08e37ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:34:58.996797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-4093e723-0f85-4dd5-b6ad-76608f084dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:59.073348 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1362/2698] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:34:58 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:34:59.084778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:59.086628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:34:59.098476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:34:59.149044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2361cd9-82c2-425c-b601-af4a995145b3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:59.153922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:59.154535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:59.154812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:59.155115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:59.166497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:59.168368 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:59.180493 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1337/2699] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:59 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:34:59.293817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4093e723-0f85-4dd5-b6ad-76608f084dea None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:34:59.308370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4093e723-0f85-4dd5-b6ad-76608f084dea None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:34:59.313005 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76d14f9-fd8a-4066-b570-a7c5bcb8aa6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:34:59.313679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76d14f9-fd8a-4066-b570-a7c5bcb8aa6d service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:34:59.313952 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76d14f9-fd8a-4066-b570-a7c5bcb8aa6d service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:34:59.314248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76d14f9-fd8a-4066-b570-a7c5bcb8aa6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:34:59.377233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76d14f9-fd8a-4066-b570-a7c5bcb8aa6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:34:59.379461 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76d14f9-fd8a-4066-b570-a7c5bcb8aa6d service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:34:59.394843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1363/2700] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:34:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:02.065921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:02.150840 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf914e1-f15c-4dc7-9dca-5050bba1cfeb service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:02.159228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:02.160776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:02.161461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:02.161960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] PATH_INFO: `/v3/users/c9726973c66448eeb588aa85e9e18b4a` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:02.163657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9726973c66448eeb588aa85e9e18b4a)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:02.166586 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:02.236584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9726973c66448eeb588aa85e9e18b4a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:02.246515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9726973c66448eeb588aa85e9e18b4a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:02.255961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9726973c66448eeb588aa85e9e18b4a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:02.256700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9726973c66448eeb588aa85e9e18b4a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:02.257355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9726973c66448eeb588aa85e9e18b4a'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:02.278446 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1338/2701] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:02 2022] DELETE /identity/v3/users/c9726973c66448eeb588aa85e9e18b4a => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:02.283696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0191897d-d46b-449a-9834-93ffb893228f None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:02.285302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0191897d-d46b-449a-9834-93ffb893228f None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:02.285302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0191897d-d46b-449a-9834-93ffb893228f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:02.307096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-0191897d-d46b-449a-9834-93ffb893228f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:02.308249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0191897d-d46b-449a-9834-93ffb893228f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:02.365516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:02.395574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1364/2702] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:02.408355 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:02.409934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:02.451876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:02.490219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d464b628-0a4d-407d-a65a-31f21334f964 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:02.494152 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:02.494778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:02.495039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:02.495401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:02.508072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:02.509646 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:02.522852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1339/2703] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:02.877956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0191897d-d46b-449a-9834-93ffb893228f None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:02.880145 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0191897d-d46b-449a-9834-93ffb893228f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:02.900300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0191897d-d46b-449a-9834-93ffb893228f None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:02.909295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:02.910050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:02.910569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:02.911029 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] PATH_INFO: `/v3/projects/bad6527e51174df28f58ce373e0d9651` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:02.912482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=bad6527e51174df28f58ce373e0d9651)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:02.914885 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:02.990251 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bad6527e51174df28f58ce373e0d9651', 'request_id': 'req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:02.997921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bad6527e51174df28f58ce373e0d9651', 'request_id': 'req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.005341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bad6527e51174df28f58ce373e0d9651', 'request_id': 'req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.024817 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] The token cache is being invalidate because project bad6527e51174df28f58ce373e0d9651 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:03.025304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.025977 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.028083 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1365/2704] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:02 2022] DELETE /identity/v3/projects/bad6527e51174df28f58ce373e0d9651 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:03.032909 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:03.043526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e381841-d7bd-409c-a1db-1efa27d23541 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:03.117818 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:03.118682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:03.118838 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:03.118996 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] PATH_INFO: `/v3/users/75bdf07ecad140c6aff3ae32a08e37ec` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:03.120192 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] RBAC: Authorizing `identity:delete_user(user_id=75bdf07ecad140c6aff3ae32a08e37ec)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:03.122234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:03.196840 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75bdf07ecad140c6aff3ae32a08e37ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.209717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75bdf07ecad140c6aff3ae32a08e37ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.219676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75bdf07ecad140c6aff3ae32a08e37ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.220313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75bdf07ecad140c6aff3ae32a08e37ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.221046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0164d778-4222-47ec-90b4-4d36efa1e39e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75bdf07ecad140c6aff3ae32a08e37ec'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.243789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1340/2705] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:03 2022] DELETE /identity/v3/users/75bdf07ecad140c6aff3ae32a08e37ec => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:03.558997 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:03.560716 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:03.575219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cf0c34-db68-4ee2-8d3d-4d9e8c4684ab admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:03.639736 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:03.640377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:03.640790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:03.641170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] PATH_INFO: `/v3/projects/76971985b5aa41f2aabf1a7a1a37dd13` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:03.642377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76971985b5aa41f2aabf1a7a1a37dd13)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:03.644359 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:03.709300 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76971985b5aa41f2aabf1a7a1a37dd13', 'request_id': 'req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.716142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76971985b5aa41f2aabf1a7a1a37dd13', 'request_id': 'req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.723279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76971985b5aa41f2aabf1a7a1a37dd13', 'request_id': 'req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.727502 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:03.743142 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] The token cache is being invalidate because project 76971985b5aa41f2aabf1a7a1a37dd13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:03.743544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.744583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2dfa21a4-e7f8-4daa-a971-58a65c0c8d19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:03.746660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1366/2706] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:03 2022] DELETE /identity/v3/projects/76971985b5aa41f2aabf1a7a1a37dd13 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:03.753065 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15ccc12-8fa7-47f4-a0a7-1ce0db44633c None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:03.753395 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15ccc12-8fa7-47f4-a0a7-1ce0db44633c None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:03.753667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15ccc12-8fa7-47f4-a0a7-1ce0db44633c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:03.775962 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c15ccc12-8fa7-47f4-a0a7-1ce0db44633c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:03.777238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c15ccc12-8fa7-47f4-a0a7-1ce0db44633c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:03.829297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:03.845339 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1341/2707] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:35:03.855465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd221b3f-e7fd-414b-bb79-9b9148e345a4 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:03.855737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd221b3f-e7fd-414b-bb79-9b9148e345a4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:03.856003 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd221b3f-e7fd-414b-bb79-9b9148e345a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:03.878376 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-cd221b3f-e7fd-414b-bb79-9b9148e345a4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:03.879606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cd221b3f-e7fd-414b-bb79-9b9148e345a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:03.888082 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1367/2708] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:03.892865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15ccc12-8fa7-47f4-a0a7-1ce0db44633c None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:03.903455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15ccc12-8fa7-47f4-a0a7-1ce0db44633c None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:03.909021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:03.909661 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:03.909932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:03.910199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:03.910879 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:35:03.911180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:03.912785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:03.926198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1342/2709] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:35:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:35:03.933600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd221b3f-e7fd-414b-bb79-9b9148e345a4 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:03.944495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd221b3f-e7fd-414b-bb79-9b9148e345a4 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:03.951162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:03.951944 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:03.952344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:03.952748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:03.953667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:03.956712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:03.996353 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1368/2710] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:03 2022] POST /identity/v3/projects => generated 387 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:04.001300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.012165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53c34b-3df5-4363-a508-c136a4559101 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.086060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.086411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.086712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.086975 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.087664 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.089830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.115068 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1343/2711] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:03 2022] POST /identity/v3/users => generated 294 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:04.119979 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.132069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f584c8f-d775-4c56-9938-e4fc352773c6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.136907 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.137661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.137934 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.138198 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.138837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.140431 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.150705 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1369/2712] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:04.154949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.165589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d90f3-e703-4fac-957e-b67844c5ca90 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.172022 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2d694e-4710-401e-8c0e-ffcd1e18e7d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.172656 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2d694e-4710-401e-8c0e-ffcd1e18e7d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.173087 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2d694e-4710-401e-8c0e-ffcd1e18e7d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.173516 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2d694e-4710-401e-8c0e-ffcd1e18e7d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.174543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2d694e-4710-401e-8c0e-ffcd1e18e7d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.177189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2d694e-4710-401e-8c0e-ffcd1e18e7d2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.194205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1344/2713] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:35:04.200320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.211573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec0df9a-7cb8-4143-b6d0-367df4f63d8c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.217123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.217850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.218109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.218393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] PATH_INFO: `/v3/projects/a08c7ea74b3f4be0bdb40f1fd777315c/users/963a3199eeeb4665bb42f773619a4ea9/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.241094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a08c7ea74b3f4be0bdb40f1fd777315c, user_id=963a3199eeeb4665bb42f773619a4ea9, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.244170 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.259370 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1370/2714] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:04 2022] PUT /identity/v3/projects/a08c7ea74b3f4be0bdb40f1fd777315c/users/963a3199eeeb4665bb42f773619a4ea9/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:04.270986 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead06532-f2f7-4ac1-8914-06d9c2d49612 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.271263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead06532-f2f7-4ac1-8914-06d9c2d49612 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.271543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead06532-f2f7-4ac1-8914-06d9c2d49612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.305140 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ead06532-f2f7-4ac1-8914-06d9c2d49612 None None] MFA Rules not processed for user `963a3199eeeb4665bb42f773619a4ea9`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:04.306390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ead06532-f2f7-4ac1-8914-06d9c2d49612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:04.377675 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1345/2715] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:04.382524 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.449508 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b008d4-fcc9-4ba6-bd5e-19d7455fa024 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.455338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.455980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.456256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.457304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.457304 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.459094 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.498995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1371/2716] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:04 2022] POST /identity/v3/projects => generated 387 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:04.503891 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead06532-f2f7-4ac1-8914-06d9c2d49612 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.518813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead06532-f2f7-4ac1-8914-06d9c2d49612 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.574370 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:04.589523 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.590139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.590412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.590676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.591405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.593366 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.621747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1346/2717] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:04 2022] POST /identity/v3/users => generated 300 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:35:04.623002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.633935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085753f-9e19-41c6-a631-b05cbf05255d admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.639075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.639671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.639947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.640212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.640866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.642731 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.654506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1372/2718] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:04.659620 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.670295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed00c8-3a9c-441f-af7b-7523d4233f04 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.675747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.676398 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.676785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.677052 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] PATH_INFO: `/v3/projects/7a8922327e8844f498e4a771187ef593/users/2d3971f8eb8e41ef8b8ccb76f8418598/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.693710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a8922327e8844f498e4a771187ef593, user_id=2d3971f8eb8e41ef8b8ccb76f8418598, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.696469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.712740 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1347/2719] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:04 2022] PUT /identity/v3/projects/7a8922327e8844f498e4a771187ef593/users/2d3971f8eb8e41ef8b8ccb76f8418598/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:04.717136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.728184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11e4f4a-390c-4963-9e9a-7f9528f2cecd admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.793663 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eac78c-76c0-4010-abda-b18710484b60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.794201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eac78c-76c0-4010-abda-b18710484b60 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.794481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eac78c-76c0-4010-abda-b18710484b60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.794748 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eac78c-76c0-4010-abda-b18710484b60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.795417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44eac78c-76c0-4010-abda-b18710484b60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.797039 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44eac78c-76c0-4010-abda-b18710484b60 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.807750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1373/2720] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:04.812687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:04.829449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf5779-7667-47cd-a240-ab306abc7ab9 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:04.835737 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:04.836402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.836671 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.836934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] PATH_INFO: `/v3/projects/7a8922327e8844f498e4a771187ef593/users/2d3971f8eb8e41ef8b8ccb76f8418598/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.839231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a8922327e8844f498e4a771187ef593, user_id=2d3971f8eb8e41ef8b8ccb76f8418598, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:04.841957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:04.854617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1348/2721] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:04 2022] PUT /identity/v3/projects/7a8922327e8844f498e4a771187ef593/users/2d3971f8eb8e41ef8b8ccb76f8418598/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:04.869527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a00d6cb-9bdf-482c-b1b1-3f86838e8193 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:04.869886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a00d6cb-9bdf-482c-b1b1-3f86838e8193 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:04.870230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a00d6cb-9bdf-482c-b1b1-3f86838e8193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:04.880315 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:04.910113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1a00d6cb-9bdf-482c-b1b1-3f86838e8193 None None] MFA Rules not processed for user `2d3971f8eb8e41ef8b8ccb76f8418598`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:04.911356 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1a00d6cb-9bdf-482c-b1b1-3f86838e8193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:04.987899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1374/2722] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:04 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:05.003026 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:05.005000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:05.064924 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd61c05-1cf8-48af-b9a1-2a169cfcf30a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:05.069654 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:05.070255 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:05.070524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:05.070785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:05.081201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:05.082690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:05.092766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1349/2723] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:05 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:05.823320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a00d6cb-9bdf-482c-b1b1-3f86838e8193 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:05.894168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a00d6cb-9bdf-482c-b1b1-3f86838e8193 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:05.899090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:05.899753 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:05.900009 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:05.900249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] PATH_INFO: `/v3/users/963a3199eeeb4665bb42f773619a4ea9` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:05.901473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] RBAC: Authorizing `identity:delete_user(user_id=963a3199eeeb4665bb42f773619a4ea9)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:05.903406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:05.966151 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '963a3199eeeb4665bb42f773619a4ea9'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:05.982910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '963a3199eeeb4665bb42f773619a4ea9'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:05.984202 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '963a3199eeeb4665bb42f773619a4ea9'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:05.994942 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '963a3199eeeb4665bb42f773619a4ea9'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.003165 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-db678283-acbc-42db-9e7d-3e5208a47820 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '963a3199eeeb4665bb42f773619a4ea9'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.011025 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1375/2724] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:05 2022] DELETE /identity/v3/users/963a3199eeeb4665bb42f773619a4ea9 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:06.014569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:06.015233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:06.015516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:06.015787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:06.024668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ba2fea-1a2c-44ed-b710-7e66b2f95b91 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:06.029488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:06.035914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:06.038583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:06.079758 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:06.080439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:06.080856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:06.081489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:06.084011 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:06.108789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1376/2725] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:35:06.120710 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:06.122241 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:06.161439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:06.182444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:06.182883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3868fa-46e1-428d-96a6-439f95e48fdb None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:06.183910 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:06.186687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:06.187430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:06.187720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:06.187982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:06.198481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:06.199923 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:06.204038 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1350/2726] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:06 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:06.210058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1377/2727] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:06.508369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:06.510115 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:06.530169 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564116e1-0f11-4b55-b58a-519fc7b30720 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:06.537389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:06.537982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:06.538256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:06.538531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] PATH_INFO: `/v3/projects/a08c7ea74b3f4be0bdb40f1fd777315c` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:06.539904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a08c7ea74b3f4be0bdb40f1fd777315c)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:06.541821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:06.595548 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a08c7ea74b3f4be0bdb40f1fd777315c', 'request_id': 'req-649c4196-c136-4c3b-8bbe-fbab54715c7f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.604513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a08c7ea74b3f4be0bdb40f1fd777315c', 'request_id': 'req-649c4196-c136-4c3b-8bbe-fbab54715c7f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.611515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a08c7ea74b3f4be0bdb40f1fd777315c', 'request_id': 'req-649c4196-c136-4c3b-8bbe-fbab54715c7f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.623756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] The token cache is being invalidate because project a08c7ea74b3f4be0bdb40f1fd777315c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:06.624050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.624723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.626333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1351/2728] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:06 2022] DELETE /identity/v3/projects/a08c7ea74b3f4be0bdb40f1fd777315c => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:06.630530 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:06.640850 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf0b39-d955-4f58-9d4b-1c33562e2afb service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:06.704642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:06.705491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:06.705797 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:06.706090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] PATH_INFO: `/v3/users/2d3971f8eb8e41ef8b8ccb76f8418598` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:06.707292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d3971f8eb8e41ef8b8ccb76f8418598)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:06.709253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:06.764616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d3971f8eb8e41ef8b8ccb76f8418598'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.770036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:06.780276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d3971f8eb8e41ef8b8ccb76f8418598'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.781013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d3971f8eb8e41ef8b8ccb76f8418598'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.790427 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d3971f8eb8e41ef8b8ccb76f8418598'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.797749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-244f44ab-4eaf-4e53-a892-53d302e7a7e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d3971f8eb8e41ef8b8ccb76f8418598'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:06.804017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1378/2729] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:06 2022] DELETE /identity/v3/users/2d3971f8eb8e41ef8b8ccb76f8418598 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:07.107619 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.109163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:07.123960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649c4196-c136-4c3b-8bbe-fbab54715c7f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.195019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.195810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.196088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.196367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] PATH_INFO: `/v3/projects/7a8922327e8844f498e4a771187ef593` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.197634 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a8922327e8844f498e4a771187ef593)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:07.199537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:07.253093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a8922327e8844f498e4a771187ef593', 'request_id': 'req-0a4a0aee-7609-4583-9426-e0e5a27fdb69'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:07.257046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:07.262957 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a8922327e8844f498e4a771187ef593', 'request_id': 'req-0a4a0aee-7609-4583-9426-e0e5a27fdb69'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:07.272078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a8922327e8844f498e4a771187ef593', 'request_id': 'req-0a4a0aee-7609-4583-9426-e0e5a27fdb69'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:07.285963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] The token cache is being invalidate because project 7a8922327e8844f498e4a771187ef593 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:07.286252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:07.286952 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-0a4a0aee-7609-4583-9426-e0e5a27fdb69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:07.288474 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1352/2730] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:07 2022] DELETE /identity/v3/projects/7a8922327e8844f498e4a771187ef593 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:07.293880 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca7cce-45dd-4a28-a5dd-9931191a51a6 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.294460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca7cce-45dd-4a28-a5dd-9931191a51a6 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.294460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca7cce-45dd-4a28-a5dd-9931191a51a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.316440 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-25ca7cce-45dd-4a28-a5dd-9931191a51a6 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:07.317712 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-25ca7cce-45dd-4a28-a5dd-9931191a51a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:07.382832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1379/2731] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:07.391134 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab29bc1-ee43-4ace-bff3-a12aa708af0b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.391400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab29bc1-ee43-4ace-bff3-a12aa708af0b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.391702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab29bc1-ee43-4ace-bff3-a12aa708af0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.413889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-2ab29bc1-ee43-4ace-bff3-a12aa708af0b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:07.415090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2ab29bc1-ee43-4ace-bff3-a12aa708af0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:07.426831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1353/2732] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:07.431395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ca7cce-45dd-4a28-a5dd-9931191a51a6 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.442623 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ca7cce-45dd-4a28-a5dd-9931191a51a6 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.448155 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.448793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.449062 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.449354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.450032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:35:07.450329 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:07.451935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:07.467846 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1380/2733] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:35:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:35:07.473182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab29bc1-ee43-4ace-bff3-a12aa708af0b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.487230 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab29bc1-ee43-4ace-bff3-a12aa708af0b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.493638 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.494561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.494866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.495216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.496013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:07.498900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:07.535081 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1354/2734] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:07 2022] POST /identity/v3/projects => generated 381 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:07.541020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.552379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb07c0d-8063-4ea7-91f4-124ec08bb40a admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.627309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.628485 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.628767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.629037 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.629735 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:07.631683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:07.657746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1381/2735] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:07 2022] POST /identity/v3/users => generated 291 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:35:07.663286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.674940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d47da66-b8da-490a-9468-0a4d6f120d92 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.680484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.681100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.681429 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.681677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.682379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:07.684045 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:07.695409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1355/2736] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:07.699647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.710876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4c864b-4d0e-4617-bae0-92046600a361 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.724596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6e7e76-5a6a-45f6-a9f7-5db8b0e44a0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.725470 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e7e76-5a6a-45f6-a9f7-5db8b0e44a0b admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.728276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e7e76-5a6a-45f6-a9f7-5db8b0e44a0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.728276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e7e76-5a6a-45f6-a9f7-5db8b0e44a0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.728276 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6e7e76-5a6a-45f6-a9f7-5db8b0e44a0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:07.728723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6e7e76-5a6a-45f6-a9f7-5db8b0e44a0b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:07.740774 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1382/2737] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:07 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:35:07.745488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.756236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2d3398-e26e-42cf-938b-819580164b51 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.761313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.762000 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.762276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.762539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] PATH_INFO: `/v3/projects/92af0d5bc43e4887872a22d0fc70c77a/users/685e1a432d1549b9bc22be3e944aef16/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.780057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=92af0d5bc43e4887872a22d0fc70c77a, user_id=685e1a432d1549b9bc22be3e944aef16, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:07.782826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:07.801799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1356/2738] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:07 2022] PUT /identity/v3/projects/92af0d5bc43e4887872a22d0fc70c77a/users/685e1a432d1549b9bc22be3e944aef16/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:07.812451 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5954614b-1e2e-4cb9-9545-3d30bdaba0f5 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.812831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5954614b-1e2e-4cb9-9545-3d30bdaba0f5 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.813108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5954614b-1e2e-4cb9-9545-3d30bdaba0f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:07.843189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5954614b-1e2e-4cb9-9545-3d30bdaba0f5 None None] MFA Rules not processed for user `685e1a432d1549b9bc22be3e944aef16`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:07.844460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5954614b-1e2e-4cb9-9545-3d30bdaba0f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:07.869953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:07.918915 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1383/2739] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:07 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:35:07.923622 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:07.993795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cb95a2-0e0e-4d64-8afb-084af0122f0a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:07.998813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:07.999424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:07.999697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:07.999964 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.000610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:08.002533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:08.047644 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1357/2740] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:07 2022] POST /identity/v3/projects => generated 379 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:08.053437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5954614b-1e2e-4cb9-9545-3d30bdaba0f5 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:08.065135 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5954614b-1e2e-4cb9-9545-3d30bdaba0f5 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:08.136393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:08.136835 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:08.137110 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:08.137408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.138086 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:08.140033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:08.164204 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1384/2741] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:08 2022] POST /identity/v3/users => generated 296 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:08.169389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:08.184513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae492dd-8a26-4884-a7f2-bd7f1206fd8a admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:08.192062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:08.192819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:08.193094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:08.193388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.194072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:08.195706 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:08.207522 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1358/2742] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:08 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:35:08.212178 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:08.223863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062d9c71-7046-4c32-a0c9-508c393889ff admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:08.230491 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:08.231144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:08.231436 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:08.231711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] PATH_INFO: `/v3/projects/43a7b7694aec40f9afc381ebce644401/users/71ae5fe2fc8c4f0899eb386e73177366/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.249410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43a7b7694aec40f9afc381ebce644401, user_id=71ae5fe2fc8c4f0899eb386e73177366, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:08.252136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:08.272709 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1385/2743] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:08 2022] PUT /identity/v3/projects/43a7b7694aec40f9afc381ebce644401/users/71ae5fe2fc8c4f0899eb386e73177366/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:35:08.278451 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:08.291364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fea690-ed38-4ab2-8652-aab90db296c9 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:08.350798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:08.362691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8c24b2-92b3-4e7c-9571-a2855c757e14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:08.363609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8c24b2-92b3-4e7c-9571-a2855c757e14 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:08.363976 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8c24b2-92b3-4e7c-9571-a2855c757e14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:08.364602 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8c24b2-92b3-4e7c-9571-a2855c757e14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.365384 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8c24b2-92b3-4e7c-9571-a2855c757e14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:08.367310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8c24b2-92b3-4e7c-9571-a2855c757e14 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:08.377940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1359/2744] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:08 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:08.382762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:08.395123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e68ed7-cc68-44f6-be94-8c3eb51ad5e3 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:08.400384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:08.401113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:08.401478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:08.401943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] PATH_INFO: `/v3/projects/43a7b7694aec40f9afc381ebce644401/users/71ae5fe2fc8c4f0899eb386e73177366/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.404121 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43a7b7694aec40f9afc381ebce644401, user_id=71ae5fe2fc8c4f0899eb386e73177366, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:08.406945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:08.419279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1386/2745] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:08 2022] PUT /identity/v3/projects/43a7b7694aec40f9afc381ebce644401/users/71ae5fe2fc8c4f0899eb386e73177366/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:08.431921 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b6b97-cf4e-4ced-b26f-2f02d2a4afc9 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:08.432205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b6b97-cf4e-4ced-b26f-2f02d2a4afc9 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:08.432467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b6b97-cf4e-4ced-b26f-2f02d2a4afc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.464868 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-955b6b97-cf4e-4ced-b26f-2f02d2a4afc9 None None] MFA Rules not processed for user `71ae5fe2fc8c4f0899eb386e73177366`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:08.466475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-955b6b97-cf4e-4ced-b26f-2f02d2a4afc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:08.541326 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1360/2746] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:08 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:08.553164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:08.555090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:08.618288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8173c0c2-a0a7-436e-87d9-02bbe0c8cbf2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:08.621956 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe78cec-8a65-47c3-a586-7fe9bcce1d6f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:08.622609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe78cec-8a65-47c3-a586-7fe9bcce1d6f service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:08.622898 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe78cec-8a65-47c3-a586-7fe9bcce1d6f service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:08.623166 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe78cec-8a65-47c3-a586-7fe9bcce1d6f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:08.632671 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe78cec-8a65-47c3-a586-7fe9bcce1d6f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:08.634201 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe78cec-8a65-47c3-a586-7fe9bcce1d6f service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:08.644495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1387/2747] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:08 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:09.099416 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955b6b97-cf4e-4ced-b26f-2f02d2a4afc9 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:09.185100 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955b6b97-cf4e-4ced-b26f-2f02d2a4afc9 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:09.191319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:09.192005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:09.192385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:09.192649 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] PATH_INFO: `/v3/users/685e1a432d1549b9bc22be3e944aef16` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:09.194501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] RBAC: Authorizing `identity:delete_user(user_id=685e1a432d1549b9bc22be3e944aef16)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:09.197227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:09.266443 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '685e1a432d1549b9bc22be3e944aef16'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:09.275610 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '685e1a432d1549b9bc22be3e944aef16'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:09.298789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '685e1a432d1549b9bc22be3e944aef16'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:09.299697 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '685e1a432d1549b9bc22be3e944aef16'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:09.300723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '685e1a432d1549b9bc22be3e944aef16'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:09.318823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:09.329253 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1361/2748] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:09 2022] DELETE /identity/v3/users/685e1a432d1549b9bc22be3e944aef16 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:09.332035 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:09.332592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:09.335707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:09.355120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:09.356487 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:09.371000 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:09.432179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1388/2749] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:09.444405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:09.446272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:09.514208 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfc6fc0-3b14-4162-8491-fcd1fac82364 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:09.518279 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:09.519281 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:09.519717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:09.520391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:09.531037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:09.532496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:09.543055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1362/2750] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:09.618395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:09.620917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:09.812763 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eb656c-dcee-47d1-9e03-86da85b097b7 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:09.822284 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:09.823004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:09.823280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:09.823546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:09.826378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:09.846144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:09.847953 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:09.860604 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1389/2751] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:09.952950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:09.954478 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:09.981515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee92dda-96cd-4d49-9a21-ad6683320dad service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:09.988831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:09.989942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:09.990216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:09.990469 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] PATH_INFO: `/v3/projects/92af0d5bc43e4887872a22d0fc70c77a` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:09.991669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] RBAC: Authorizing `identity:delete_project(project_id=92af0d5bc43e4887872a22d0fc70c77a)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:09.993669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:10.044904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92af0d5bc43e4887872a22d0fc70c77a', 'request_id': 'req-500122be-1157-49bb-815a-fd5a0ef51012'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.054347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92af0d5bc43e4887872a22d0fc70c77a', 'request_id': 'req-500122be-1157-49bb-815a-fd5a0ef51012'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.062222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92af0d5bc43e4887872a22d0fc70c77a', 'request_id': 'req-500122be-1157-49bb-815a-fd5a0ef51012'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.073428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] The token cache is being invalidate because project 92af0d5bc43e4887872a22d0fc70c77a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:10.073694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.075059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.076578 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1363/2752] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:09 2022] DELETE /identity/v3/projects/92af0d5bc43e4887872a22d0fc70c77a => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:10.080908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:10.097482 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d5eebe-987a-4257-bc0f-86c526214763 service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:10.169055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:10.170092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:10.170465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:10.170770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] PATH_INFO: `/v3/users/71ae5fe2fc8c4f0899eb386e73177366` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:10.172435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] RBAC: Authorizing `identity:delete_user(user_id=71ae5fe2fc8c4f0899eb386e73177366)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:10.174831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:10.235183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71ae5fe2fc8c4f0899eb386e73177366'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.236454 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:10.262457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71ae5fe2fc8c4f0899eb386e73177366'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.263512 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71ae5fe2fc8c4f0899eb386e73177366'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.274932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71ae5fe2fc8c4f0899eb386e73177366'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.288391 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f70977bf-b416-4ea9-9fd5-0d1232bbeebe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71ae5fe2fc8c4f0899eb386e73177366'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.296906 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1390/2753] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:10 2022] DELETE /identity/v3/users/71ae5fe2fc8c4f0899eb386e73177366 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:10.615908 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:10.617636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:10.638242 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500122be-1157-49bb-815a-fd5a0ef51012 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:10.714539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:10.715225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:10.715506 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:10.715842 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] PATH_INFO: `/v3/projects/43a7b7694aec40f9afc381ebce644401` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:10.717436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43a7b7694aec40f9afc381ebce644401)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:10.719454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:10.749553 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:10.780570 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43a7b7694aec40f9afc381ebce644401', 'request_id': 'req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.789111 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43a7b7694aec40f9afc381ebce644401', 'request_id': 'req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.796755 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43a7b7694aec40f9afc381ebce644401', 'request_id': 'req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.810472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] The token cache is being invalidate because project 43a7b7694aec40f9afc381ebce644401 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:10.810747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.811535 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-96d9f7d6-c65c-4a98-bd57-1a67c05aecb2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:10.813181 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1364/2754] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:10 2022] DELETE /identity/v3/projects/43a7b7694aec40f9afc381ebce644401 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:10.818789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289bc4c9-9c9b-42cf-922d-1807a4d8d6fa None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:10.819320 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289bc4c9-9c9b-42cf-922d-1807a4d8d6fa None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:10.819426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289bc4c9-9c9b-42cf-922d-1807a4d8d6fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:10.842837 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-289bc4c9-9c9b-42cf-922d-1807a4d8d6fa None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:10.844397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-289bc4c9-9c9b-42cf-922d-1807a4d8d6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:10.916839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1391/2755] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:10.925138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25908b5-247d-42c8-ad00-20016303a1fb None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:10.925424 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25908b5-247d-42c8-ad00-20016303a1fb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:10.925685 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25908b5-247d-42c8-ad00-20016303a1fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:10.946757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-c25908b5-247d-42c8-ad00-20016303a1fb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:10.947901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-c25908b5-247d-42c8-ad00-20016303a1fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:10.955994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1365/2756] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:10.960680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289bc4c9-9c9b-42cf-922d-1807a4d8d6fa None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:10.973031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289bc4c9-9c9b-42cf-922d-1807a4d8d6fa None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:10.979405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:10.980028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:10.980299 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:10.980560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:10.981236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:35:10.981541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:10.983133 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:10.996386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1392/2757] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:35:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:35:11.000930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25908b5-247d-42c8-ad00-20016303a1fb None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:11.011005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25908b5-247d-42c8-ad00-20016303a1fb None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:11.017275 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:11.018072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:11.018350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:11.018614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:11.019278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:11.021533 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:11.053015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1366/2758] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:10 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:11.057515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:11.068752 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d473d0-a0f4-4166-94f0-1c89f3ac9cfe admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:11.131704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:11.132309 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:11.132583 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:11.132849 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:11.133542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:11.135438 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:11.157654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1393/2759] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:11 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:11.162711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:11.173565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dc8f63-f179-42ce-b830-c1bc6e1752c4 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:11.178847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:11.179435 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:11.179712 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:11.179966 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:11.180666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:11.182293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:11.192692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1367/2760] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:11.197542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:11.207772 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3616def-ca7f-415c-a4b3-0a2664e3dcbf admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:11.212630 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee1810b-5bbc-49ec-aaa8-b1293b0bffa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:11.213254 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1810b-5bbc-49ec-aaa8-b1293b0bffa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:11.213529 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1810b-5bbc-49ec-aaa8-b1293b0bffa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:11.213813 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1810b-5bbc-49ec-aaa8-b1293b0bffa5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:11.214507 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee1810b-5bbc-49ec-aaa8-b1293b0bffa5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:11.216095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee1810b-5bbc-49ec-aaa8-b1293b0bffa5 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:11.226936 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1394/2761] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:35:11.231887 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:11.242315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6242479d-1a81-40ea-baaf-2c160532db96 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:11.248149 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:11.248795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:11.249062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:11.249353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] PATH_INFO: `/v3/projects/80ccce2a519b40748805db563d825564/users/48390188a6e84ef18b63ff4ea4994284/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:11.265350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=80ccce2a519b40748805db563d825564, user_id=48390188a6e84ef18b63ff4ea4994284, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:11.268031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:11.281733 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1368/2762] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:11 2022] PUT /identity/v3/projects/80ccce2a519b40748805db563d825564/users/48390188a6e84ef18b63ff4ea4994284/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:11.292940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:11.292940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:11.293291 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:11.324202 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] MFA Rules not processed for user `48390188a6e84ef18b63ff4ea4994284`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:11.325481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:11.344576 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:11.398624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1395/2763] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:11 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:11.407220 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:11.408932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:11.477221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb42bab0-59b1-40c6-8fdf-9bc5ec498d3f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:11.481036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:11.481725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:11.482004 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:11.482263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:11.491043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:11.492489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:11.502019 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1369/2764] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:11 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:17.763539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:17.765150 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:17.826700 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b90a5e-9f20-4ce5-ae93-b0207af83fb0 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:17.830533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:17.831153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:17.831422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:17.831686 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:17.833608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:17.899581 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:17.901056 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:17.911864 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1396/2765] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:21.156577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:21.158461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:21.177160 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b4e293-00b7-4705-9df8-ca1074e304e3 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:21.184234 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:21.185015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.185389 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.185731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] PATH_INFO: `/v3/users/d750e6e172d34d4a9333e0c059c5442b` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.187173 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d750e6e172d34d4a9333e0c059c5442b)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:21.189333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:21.226157 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:21.242910 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a850876a-ed5c-4f79-8084-6e9263a2d042 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:21.249294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:21.250637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.250965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.251479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] PATH_INFO: `/v3/users/48390188a6e84ef18b63ff4ea4994284` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.252941 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] RBAC: Authorizing `identity:delete_user(user_id=48390188a6e84ef18b63ff4ea4994284)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:21.254566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:21.257766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd750e6e172d34d4a9333e0c059c5442b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.258701 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:21.269369 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd750e6e172d34d4a9333e0c059c5442b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.281257 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd750e6e172d34d4a9333e0c059c5442b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.282252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd750e6e172d34d4a9333e0c059c5442b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.283341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-58502010-42be-411a-a63a-8fd62927ff6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd750e6e172d34d4a9333e0c059c5442b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.317260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1370/2766] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:21 2022] DELETE /identity/v3/users/d750e6e172d34d4a9333e0c059c5442b => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:21.332013 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48390188a6e84ef18b63ff4ea4994284'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.346032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48390188a6e84ef18b63ff4ea4994284'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.346831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48390188a6e84ef18b63ff4ea4994284'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.349015 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:21.358233 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48390188a6e84ef18b63ff4ea4994284'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.367999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48390188a6e84ef18b63ff4ea4994284'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.375756 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1397/2767] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:35:21 2022] DELETE /identity/v3/users/48390188a6e84ef18b63ff4ea4994284 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:35:21.382063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.382717 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.383150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.406356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:21.407607 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:21.487564 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1371/2768] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:21.498720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:21.500283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:21.550201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:21.552475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:21.570723 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e54db3f-0c8a-4c2a-964f-ef86a50d9ac7 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:21.574582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f18c37-33be-4b19-ab1b-48958bc4b719 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:21.577597 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:21.578289 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860c7449-04c3-4ccb-ae70-b28ca55f5ed4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:21.578929 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860c7449-04c3-4ccb-ae70-b28ca55f5ed4 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.579211 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860c7449-04c3-4ccb-ae70-b28ca55f5ed4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.579574 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860c7449-04c3-4ccb-ae70-b28ca55f5ed4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.579874 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.580191 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.580458 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] PATH_INFO: `/v3/projects/c938ae5fd22a43e8a01859da2a336b53` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.583015 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c938ae5fd22a43e8a01859da2a336b53)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:21.585517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:21.590999 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860c7449-04c3-4ccb-ae70-b28ca55f5ed4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:21.592456 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860c7449-04c3-4ccb-ae70-b28ca55f5ed4 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:21.604008 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1398/2769] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:21.637186 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c938ae5fd22a43e8a01859da2a336b53', 'request_id': 'req-66551869-fa5a-4566-901c-d7e68a5f7042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.648062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c938ae5fd22a43e8a01859da2a336b53', 'request_id': 'req-66551869-fa5a-4566-901c-d7e68a5f7042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.656621 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c938ae5fd22a43e8a01859da2a336b53', 'request_id': 'req-66551869-fa5a-4566-901c-d7e68a5f7042'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.672227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] The token cache is being invalidate because project c938ae5fd22a43e8a01859da2a336b53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:21.672524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.673229 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-66551869-fa5a-4566-901c-d7e68a5f7042 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:21.674813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1372/2770] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:21 2022] DELETE /identity/v3/projects/c938ae5fd22a43e8a01859da2a336b53 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:35:21.681760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225ed78e-ea74-4be6-9b8c-a96ad2076c6e None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.682050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225ed78e-ea74-4be6-9b8c-a96ad2076c6e None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.682350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225ed78e-ea74-4be6-9b8c-a96ad2076c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.705469 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-225ed78e-ea74-4be6-9b8c-a96ad2076c6e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:21.706728 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-225ed78e-ea74-4be6-9b8c-a96ad2076c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:21.781690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1399/2771] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:35:21.789799 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16703744-308b-4c75-afd8-744c900d74e8 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.790318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16703744-308b-4c75-afd8-744c900d74e8 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.790778 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16703744-308b-4c75-afd8-744c900d74e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.815264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-16703744-308b-4c75-afd8-744c900d74e8 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:21.817642 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-16703744-308b-4c75-afd8-744c900d74e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:21.827541 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1373/2772] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:21.832146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225ed78e-ea74-4be6-9b8c-a96ad2076c6e None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:21.842723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225ed78e-ea74-4be6-9b8c-a96ad2076c6e None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:21.847425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:21.848189 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.848646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.848927 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.849625 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:35:21.849928 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:21.851533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:21.864213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1400/2773] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:35:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:35:21.868700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16703744-308b-4c75-afd8-744c900d74e8 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:21.878488 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16703744-308b-4c75-afd8-744c900d74e8 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:21.884221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:21.884898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:21.885267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:21.885557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:21.886260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:21.888217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:21.922782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1374/2774] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:21 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:35:21.924050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:21.935459 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6fa310-a8d6-4139-b7de-9182532d4292 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.007321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.007321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.007321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.007321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.007874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.010773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.035660 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1401/2775] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:21 2022] POST /identity/v3/users => generated 289 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:22.041063 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.056476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6888098-6880-4cf9-a016-3378b24652bd admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.062544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.063166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.063436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.063695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.064347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.066080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.077780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1375/2776] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:22 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:22.082566 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.088579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:22.097250 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2241af5-00c8-43bf-8086-be1d2822338f admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.105072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2078d623-37a5-4694-95c7-a50cef5cf8bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.105761 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2078d623-37a5-4694-95c7-a50cef5cf8bc admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.106033 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2078d623-37a5-4694-95c7-a50cef5cf8bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.106298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2078d623-37a5-4694-95c7-a50cef5cf8bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.106946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2078d623-37a5-4694-95c7-a50cef5cf8bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.108650 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2078d623-37a5-4694-95c7-a50cef5cf8bc admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.126921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1402/2777] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:22 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:22.128300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.135500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:DELETE=11,keystone:SELECT=86,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:22.141337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300ba793-d513-458d-ab22-ae1f81be41a2 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.148860 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.150029 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.150719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.151222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] PATH_INFO: `/v3/projects/a8055f4326f640fc8927f5cfdef39dd3/users/ee6084fffc544dfe8f1a9a30ae06be69/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.174262 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8055f4326f640fc8927f5cfdef39dd3, user_id=ee6084fffc544dfe8f1a9a30ae06be69, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.178801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.198856 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1376/2778] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:22 2022] PUT /identity/v3/projects/a8055f4326f640fc8927f5cfdef39dd3/users/ee6084fffc544dfe8f1a9a30ae06be69/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:22.205830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.206007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.206236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.206546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.208286 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:22.234822 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:22.236120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:22.304455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823f6372-0235-4fe3-8a26-e7d90cb5d7a7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.312304 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.312965 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.313268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.313555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] PATH_INFO: `/v3/projects/80ccce2a519b40748805db563d825564` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.314878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1403/2779] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:22.319159 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80ccce2a519b40748805db563d825564)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.321147 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.325183 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.326808 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:22.383318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80ccce2a519b40748805db563d825564', 'request_id': 'req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:22.392132 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80ccce2a519b40748805db563d825564', 'request_id': 'req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:22.400013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80ccce2a519b40748805db563d825564', 'request_id': 'req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:22.400278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c476dc6-b0ef-4623-87e4-e6c5e7c71b3c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.419680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] The token cache is being invalidate because project 80ccce2a519b40748805db563d825564 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:35:22.420024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:22.420617 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7bb16a69-6cb8-4898-aaa4-31e3ad41f486 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:35:22.422206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1377/2780] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:35:22 2022] DELETE /identity/v3/projects/80ccce2a519b40748805db563d825564 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:35:22.427554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcea8fa-183d-4800-948c-e409195a0524 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.427903 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcea8fa-183d-4800-948c-e409195a0524 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.428177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcea8fa-183d-4800-948c-e409195a0524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.451427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-6dcea8fa-183d-4800-948c-e409195a0524 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:22.452662 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-6dcea8fa-183d-4800-948c-e409195a0524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:22.466787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.466787 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.467499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.467499 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.469347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:22.522641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1378/2781] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:22.537183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f457e-264e-4d33-8ed8-acfab25ce47b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.537415 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f457e-264e-4d33-8ed8-acfab25ce47b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.537545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f457e-264e-4d33-8ed8-acfab25ce47b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.562100 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.564783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.572226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f76f457e-264e-4d33-8ed8-acfab25ce47b None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:22.573496 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f76f457e-264e-4d33-8ed8-acfab25ce47b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:22.573666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:22.584325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1404/2782] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:35:22.589677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1379/2783] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:22.591217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.607484 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a13d911-c517-4a8f-aa65-8c6f04447290 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.614687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.615262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.615539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.615805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.616472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:35:22.616779 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.619471 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.637238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1405/2784] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:35:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:35:22.642222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76f457e-264e-4d33-8ed8-acfab25ce47b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.650021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:22.664970 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76f457e-264e-4d33-8ed8-acfab25ce47b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.672508 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.673763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.674073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.674529 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.676828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.679325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.716509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1380/2785] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:22 2022] POST /identity/v3/projects => generated 379 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:22.720998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.732603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3d0b37-a267-4110-bd24-f9a84a55b4de admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.810379 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.811337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.811812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.812296 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.813613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.816621 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.854802 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1406/2786] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:22 2022] POST /identity/v3/users => generated 290 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:22.856448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.867876 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f78a7-e22c-4d04-9f5e-5ef206c9a0ce admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.872794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.873450 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.873721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.873991 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.874667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.876280 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.895929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1381/2787] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:22 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:22.896673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.911283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b912da8-686a-4532-9934-e63fd8ee03a8 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.916744 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162a157a-63e9-4027-8c80-d21eed26bb85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.917386 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162a157a-63e9-4027-8c80-d21eed26bb85 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.917664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162a157a-63e9-4027-8c80-d21eed26bb85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.917940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162a157a-63e9-4027-8c80-d21eed26bb85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.918609 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162a157a-63e9-4027-8c80-d21eed26bb85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.920245 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162a157a-63e9-4027-8c80-d21eed26bb85 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:22.940093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1407/2788] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:22 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:22.942376 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:22.954468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cc645-d89e-417e-bdd4-d1bf8d9658c6 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:22.961005 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:22.961769 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:22.962085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:22.962314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] PATH_INFO: `/v3/projects/79e671941c234f36b542080d256de2c6/users/3fe5a1a037fa460eb3121e4d307542da/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:22.991886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=79e671941c234f36b542080d256de2c6, user_id=3fe5a1a037fa460eb3121e4d307542da, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:22.996049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:23.019747 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1382/2789] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:22 2022] PUT /identity/v3/projects/79e671941c234f36b542080d256de2c6/users/3fe5a1a037fa460eb3121e4d307542da/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:23.027490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2852751-576c-4f55-811c-5736a8797391 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:23.027780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2852751-576c-4f55-811c-5736a8797391 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:23.028049 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2852751-576c-4f55-811c-5736a8797391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:23.061890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-d2852751-576c-4f55-811c-5736a8797391 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:23.063208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-d2852751-576c-4f55-811c-5736a8797391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:23.139616 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1408/2790] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:23.149543 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:23.151407 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:23.214224 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f79b91d-1fc2-4739-a9eb-39f7c66bf9fc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:23.218272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:23.218948 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:23.219408 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:23.219865 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:23.234238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:23.235953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:23.248272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1383/2791] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:29.245381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8036df61-5228-419a-a998-f9000d84d589 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:29.246028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8036df61-5228-419a-a998-f9000d84d589 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:29.246028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8036df61-5228-419a-a998-f9000d84d589 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:29.283460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-8036df61-5228-419a-a998-f9000d84d589 None None] MFA Rules not processed for user `ee6084fffc544dfe8f1a9a30ae06be69`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:29.284722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8036df61-5228-419a-a998-f9000d84d589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:29.362827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:29.388305 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1409/2792] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:35:29.396595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:29.397982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:29.567036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1d822d-4dfa-4502-9748-8e9e3a766e31 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:29.580744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:29.581735 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:29.582176 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:29.582701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:29.595295 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:29.597180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:29.610618 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1384/2793] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:30.175503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225baa88-dd8e-41b2-9703-023b7bc49a32 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:30.175877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225baa88-dd8e-41b2-9703-023b7bc49a32 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:30.176167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225baa88-dd8e-41b2-9703-023b7bc49a32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:30.246940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-225baa88-dd8e-41b2-9703-023b7bc49a32 None None] MFA Rules not processed for user `3fe5a1a037fa460eb3121e4d307542da`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:30.248702 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-225baa88-dd8e-41b2-9703-023b7bc49a32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:30.369190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1410/2794] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:30 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:30.378761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:30.408381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3151a9b8-4d02-4e51-b09e-cadbe1ed3520 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:30.416138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:30.416784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:30.417267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:30.417545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:30.418404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:30.420616 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:30.462595 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1385/2795] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:35:30 2022] POST /identity/v3/projects => generated 377 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:30.466143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225baa88-dd8e-41b2-9703-023b7bc49a32 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:30.482252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225baa88-dd8e-41b2-9703-023b7bc49a32 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:30.613669 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:30.615569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:30.616221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:30.616683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:30.617866 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:30.621002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:30.655177 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1411/2796] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:35:30 2022] POST /identity/v3/users => generated 295 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:35:30.666041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:30.682207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dcad38-c124-4843-a91e-245ede3aa2ea admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:30.691794 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:30.692901 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:30.693708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:30.695766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:30.695766 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:30.696687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:30.717145 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1386/2797] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:30.719136 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:30.743572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a4370b-3af4-4380-87b6-2db185da1efb admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:30.755067 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:30.755903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:30.756279 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:30.756642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] PATH_INFO: `/v3/projects/6e8c33b5be4d43799914bbc93fe5954f/users/d8285f30bcee4d408d40a243f66a4b15/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:30.783088 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e8c33b5be4d43799914bbc93fe5954f, user_id=d8285f30bcee4d408d40a243f66a4b15, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:30.786260 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:30.808527 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1412/2798] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:30 2022] PUT /identity/v3/projects/6e8c33b5be4d43799914bbc93fe5954f/users/d8285f30bcee4d408d40a243f66a4b15/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:30.813750 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:30.836094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:30.846517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01221a66-c673-475d-a4ee-46b8453c2918 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:30.980031 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ff4e81-fd1f-41ea-8104-516aece9b90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:30.980666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff4e81-fd1f-41ea-8104-516aece9b90e admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:30.980940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff4e81-fd1f-41ea-8104-516aece9b90e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:30.981270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff4e81-fd1f-41ea-8104-516aece9b90e admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:30.982062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ff4e81-fd1f-41ea-8104-516aece9b90e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:30.983801 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ff4e81-fd1f-41ea-8104-516aece9b90e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:31.012795 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1387/2799] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:35:31.016834 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:31.030394 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4ae77-267f-4d7f-a28f-17928d01977e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:31.042124 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:31.044068 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:31.044718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:31.045274 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] PATH_INFO: `/v3/projects/6e8c33b5be4d43799914bbc93fe5954f/users/d8285f30bcee4d408d40a243f66a4b15/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:31.049393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e8c33b5be4d43799914bbc93fe5954f, user_id=d8285f30bcee4d408d40a243f66a4b15, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:31.054337 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:31.074905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1413/2800] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:35:31 2022] PUT /identity/v3/projects/6e8c33b5be4d43799914bbc93fe5954f/users/d8285f30bcee4d408d40a243f66a4b15/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:35:31.764112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575666c1-01f0-47f8-9678-addd1f5a26f3 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:31.765410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575666c1-01f0-47f8-9678-addd1f5a26f3 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:31.766245 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575666c1-01f0-47f8-9678-addd1f5a26f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:31.796904 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-575666c1-01f0-47f8-9678-addd1f5a26f3 None None] MFA Rules not processed for user `ee6084fffc544dfe8f1a9a30ae06be69`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:31.798711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-575666c1-01f0-47f8-9678-addd1f5a26f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:31.894070 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1388/2801] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:35:32.837515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:32.842288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:32.938790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:32.975292 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139ff120-15ed-45ee-b835-c06c2a28da4e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:32.988555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:32.989704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:32.990131 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:32.990408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:32.992352 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:33.023611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:33.027989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:33.063560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1414/2802] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:38.769360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7026989c-8933-4612-a735-681337b3169e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:38.770225 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7026989c-8933-4612-a735-681337b3169e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:38.770958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7026989c-8933-4612-a735-681337b3169e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:38.830311 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-7026989c-8933-4612-a735-681337b3169e None None] MFA Rules not processed for user `d8285f30bcee4d408d40a243f66a4b15`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:35:38.831831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7026989c-8933-4612-a735-681337b3169e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:38.964771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1389/2803] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:35:38 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:35:38.979831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:38.982034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:39.151706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c891d6-ccc7-4dba-b978-57835f8b6d40 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:39.159455 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd54735-f566-427f-b457-d8398f95f448 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:39.160285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd54735-f566-427f-b457-d8398f95f448 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:39.160721 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd54735-f566-427f-b457-d8398f95f448 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:39.161358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd54735-f566-427f-b457-d8398f95f448 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:39.251645 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd54735-f566-427f-b457-d8398f95f448 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:39.253365 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd54735-f566-427f-b457-d8398f95f448 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:39.271202 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1415/2804] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:38 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:48.956043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=87 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:49.199183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7026989c-8933-4612-a735-681337b3169e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:35:49.215075 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7026989c-8933-4612-a735-681337b3169e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:35:49.224128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:35:49.224781 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:35:49.225072 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:35:49.225374 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:35:49.228010 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:35:49.241259 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:35:49.243623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:35:49.255388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1390/2805] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:35:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:35:49.261892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=66 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:35:59.250575 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=6 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:36:03.354411 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:36:03.355162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:36:03.355162 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:36:03.376253 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] MFA Rules not processed for user `3fe5a1a037fa460eb3121e4d307542da`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:36:03.377526 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:36:03.385791 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1416/2806] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:36:03 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:36:03.739296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:36:03.750148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509d825-88da-4bfc-aba2-46ebaa8a1d7e service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:36:03.754350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:36:03.754994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:36:03.755264 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:36:03.755526 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:36:03.757584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:36:03.768468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:36:03.769950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:36:03.778249 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1391/2807] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:36:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:36:05.295118 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:36:05.296897 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:36:05.381819 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e987f0-8d49-4769-a04c-d64c2aca3e42 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:36:05.387917 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:36:05.388547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:36:05.388825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:36:05.389210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:36:05.395416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:36:05.403749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:36:05.405388 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:36:05.416608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1417/2808] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:36:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 27 02:36:13.774517 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=6 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:36:15.411396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=35 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:36:21.773784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:36:21.783467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacbd4b9-d90f-47a2-b2b9-a93650fb1706 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:36:21.787816 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:36:21.788430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:36:21.788696 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:36:21.788953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:36:21.799385 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:36:21.800823 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:36:21.812254 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1392/2809] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:36:21 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:36:31.806839 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=6 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:38:35.446767 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:38:35.459883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d719614-90af-4716-90e3-af37667d6545 service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:38:35.464494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:38:35.465153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:38:35.465435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:38:35.465746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:38:35.467948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:38:35.479546 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:38:35.481065 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:38:35.490416 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1418/2810] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:38:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:38:35.795409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:38:35.796982 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:38:35.861211 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bde459-78ed-4f8e-a4c4-271df5301a78 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:38:35.865484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bf2311-404b-4a03-ab72-a124d6eac3e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:38:35.866093 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf2311-404b-4a03-ab72-a124d6eac3e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:38:35.866361 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf2311-404b-4a03-ab72-a124d6eac3e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:38:35.866626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf2311-404b-4a03-ab72-a124d6eac3e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:38:35.868423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-88bf2311-404b-4a03-ab72-a124d6eac3e3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:38:35.882719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bf2311-404b-4a03-ab72-a124d6eac3e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:38:35.884148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bf2311-404b-4a03-ab72-a124d6eac3e3 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:38:35.895318 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1393/2811] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:38:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:38:38.529969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:38:38.539881 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69146773-a3db-44dc-adce-6b06bf7a1399 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:38:38.544249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:38:38.544838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:38:38.545113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:38:38.545401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:38:38.547402 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:38:38.562192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:38:38.563603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:38:38.575240 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1419/2812] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:38:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:38:45.890373 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=26 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:38:48.570054 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=12 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:38:55.282784 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba07d976-17d2-464e-82b6-0d29400a454f None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:38:55.283329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba07d976-17d2-464e-82b6-0d29400a454f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:38:55.283329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba07d976-17d2-464e-82b6-0d29400a454f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:38:55.303980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ba07d976-17d2-464e-82b6-0d29400a454f None None] MFA Rules not processed for user `3fe5a1a037fa460eb3121e4d307542da`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:38:55.305212 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ba07d976-17d2-464e-82b6-0d29400a454f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:38:55.312711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1394/2813] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:38:55 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:38:56.154259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:38:56.168958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db62a5-1399-4f3b-bc71-d54539ea6ae1 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:38:56.174443 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730832db-1ed1-4a81-bfe0-704420860cb4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:38:56.175052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730832db-1ed1-4a81-bfe0-704420860cb4 service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:38:56.175323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730832db-1ed1-4a81-bfe0-704420860cb4 service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:38:56.175588 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730832db-1ed1-4a81-bfe0-704420860cb4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:38:56.177501 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-730832db-1ed1-4a81-bfe0-704420860cb4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:38:56.190936 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730832db-1ed1-4a81-bfe0-704420860cb4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:38:56.192518 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730832db-1ed1-4a81-bfe0-704420860cb4 service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:38:56.204036 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1420/2814] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:38:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:39:05.298419 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:39:06.199109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=6 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:39:09.798189 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba07d976-17d2-464e-82b6-0d29400a454f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:39:09.816827 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba07d976-17d2-464e-82b6-0d29400a454f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:39:09.824299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:39:09.825074 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:39:09.825817 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:39:09.826178 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:39:09.828271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:39:09.846342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:39:09.849471 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:39:09.869347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1395/2815] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:39:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:39:19.860814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=6 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:40:01.924886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b204928-87f5-410a-8918-852b11cdbd29 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:40:01.925516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b204928-87f5-410a-8918-852b11cdbd29 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:40:01.925516 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b204928-87f5-410a-8918-852b11cdbd29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:40:01.945976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-8b204928-87f5-410a-8918-852b11cdbd29 None None] MFA Rules not processed for user `ee6084fffc544dfe8f1a9a30ae06be69`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:40:01.947161 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8b204928-87f5-410a-8918-852b11cdbd29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:40:01.955247 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1421/2816] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:40:01 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:40:11.940120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:41:15.110423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:41:15.142034 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb20fa8c-712b-4757-a17a-90dba1398739 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:41:15.146510 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:41:15.147122 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:41:15.147399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:41:15.147670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:41:15.174217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:41:15.175652 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:41:15.184968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1396/2817] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:41:15 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:41:25.180847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=17 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:41:26.524186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:41:26.524740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:41:26.524740 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:41:26.566796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] MFA Rules not processed for user `3fe5a1a037fa460eb3121e4d307542da`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:41:26.568697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:41:26.578494 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1422/2818] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:41:26 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:41:26.950367 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:41:26.963507 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5d0570-e6f4-4f1a-8d44-509699eb8720 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:41:26.968042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:41:26.968641 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:41:26.968911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:41:26.969205 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:41:26.976973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:41:26.978573 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:41:26.990822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1397/2819] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:41:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:41:28.041805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:41:28.043575 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:41:28.102441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b1dd99-dd97-4622-a7c0-db6edcba4a1c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:41:28.109547 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:41:28.110156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:41:28.110430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:41:28.110697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:41:28.120280 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:41:28.121904 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:41:28.131982 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1423/2820] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:41:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:41:30.311499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:41:30.327236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217cc711-54bb-45e5-b1f0-82883980ff99 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:41:30.333180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:41:30.333803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:41:30.334090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:41:30.334336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:41:30.336252 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:41:30.359857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:41:30.361772 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:41:30.374207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1398/2821] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:41:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:41:32.670572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:41:32.690114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f7e015-2bb6-430d-9212-e8a95bd351a1 service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:41:32.696019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:41:32.696629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:41:32.697080 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:41:32.697396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:41:32.706089 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:41:32.707722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:41:32.721066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1424/2822] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:41:32 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:41:40.368722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=12 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:41:42.717010 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=35 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:41:46.018539 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:41:46.046117 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac67910-ef88-40d5-a1ce-ccfe964fccf6 service nova] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:41:46.050309 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:41:46.050937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:41:46.051190 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:41:46.051461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:41:46.061265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:41:46.062998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:41:46.072771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1399/2823] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:41:46 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:41:56.069531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=10 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:05.542764 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:05.560633 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba1a3b7-4464-46f6-83eb-7a171aabfd54 service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:05.565437 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:05.566040 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:05.566316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:05.566584 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:05.650164 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:05.652319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:05.674447 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1425/2824] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:08.983064 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:08.996157 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710a08e-2635-4fac-8b18-d7596e1de8c1 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:09.003800 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:09.004476 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.004918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.005143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.007409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:09.020043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:09.021824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:09.045681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:09.047435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:09.058635 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1400/2825] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:09.097185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf45892b-3598-42c2-99de-627b493f6aa9 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:09.102266 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:09.103138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.103613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.103890 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] PATH_INFO: `/v3/users/ee6084fffc544dfe8f1a9a30ae06be69` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.105066 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee6084fffc544dfe8f1a9a30ae06be69)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:09.107412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:09.168888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee6084fffc544dfe8f1a9a30ae06be69'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.182715 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee6084fffc544dfe8f1a9a30ae06be69'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.183444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee6084fffc544dfe8f1a9a30ae06be69'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.191687 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee6084fffc544dfe8f1a9a30ae06be69'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.200027 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f19f75f5-e51e-4db7-a3a0-a22c0d55e8ae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee6084fffc544dfe8f1a9a30ae06be69'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.205901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1426/2826] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:09 2022] DELETE /identity/v3/users/ee6084fffc544dfe8f1a9a30ae06be69 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:09.395514 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:09.409390 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed15d348-bacb-4291-9f1f-7977d8bd4a93 service placement] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:09.469955 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:09.470577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.470852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.471116 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] PATH_INFO: `/v3/projects/a8055f4326f640fc8927f5cfdef39dd3` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.472283 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8055f4326f640fc8927f5cfdef39dd3)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:09.474196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:09.531763 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8055f4326f640fc8927f5cfdef39dd3', 'request_id': 'req-a835501e-949e-4f1f-a0e8-183d3846034f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.544402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8055f4326f640fc8927f5cfdef39dd3', 'request_id': 'req-a835501e-949e-4f1f-a0e8-183d3846034f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.551813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8055f4326f640fc8927f5cfdef39dd3', 'request_id': 'req-a835501e-949e-4f1f-a0e8-183d3846034f'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.564430 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] The token cache is being invalidate because project a8055f4326f640fc8927f5cfdef39dd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81746) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:09.564719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.565489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-a835501e-949e-4f1f-a0e8-183d3846034f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:09.567124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1401/2827] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:09 2022] DELETE /identity/v3/projects/a8055f4326f640fc8927f5cfdef39dd3 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:09.572452 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0012665c-5bac-44ba-8859-aaec7979bfdf None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.572737 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0012665c-5bac-44ba-8859-aaec7979bfdf None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.573063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0012665c-5bac-44ba-8859-aaec7979bfdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.633441 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:09.636607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-0012665c-5bac-44ba-8859-aaec7979bfdf None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:09.637876 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-0012665c-5bac-44ba-8859-aaec7979bfdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:09.706560 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1427/2828] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:09.714660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54ae28-1fb5-44ed-83d3-87347808567e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.714943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54ae28-1fb5-44ed-83d3-87347808567e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.715207 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54ae28-1fb5-44ed-83d3-87347808567e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.744277 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-ed54ae28-1fb5-44ed-83d3-87347808567e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:09.745544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-ed54ae28-1fb5-44ed-83d3-87347808567e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:09.755672 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1402/2829] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:09.760539 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0012665c-5bac-44ba-8859-aaec7979bfdf None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:09.771196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0012665c-5bac-44ba-8859-aaec7979bfdf None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:09.777383 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:09.777995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.778270 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.778538 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] PATH_INFO: `/v3/domains` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.779195 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81745) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:42:09.779496 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:09.781224 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:09.793785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1428/2830] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:42:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:42:09.800138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54ae28-1fb5-44ed-83d3-87347808567e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:09.810340 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54ae28-1fb5-44ed-83d3-87347808567e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:09.816013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:09.816598 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.816867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.817163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.817783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:09.819930 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:09.851235 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1403/2831] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:09 2022] POST /identity/v3/projects => generated 405 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:09.857277 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:09.868012 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3671f278-e69d-4782-9556-66ef2e168f9d admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:09.937969 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:09.938874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.939209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.939612 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.940636 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:09.943786 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:09.968658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1429/2832] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:09 2022] POST /identity/v3/users => generated 303 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:42:09.973138 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:09.986680 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490faf4b-415e-46f5-9410-4e1a795daf96 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:09.992143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:09.992744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:09.993037 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:09.993314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:09.993974 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:09.995569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:10.008124 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1404/2833] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:09 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:10.012711 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:10.024641 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb80be9-9618-490d-921e-306d4ea0d040 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:10.031647 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe24e93-2377-4886-8f7f-b345bc5e1122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:10.032551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe24e93-2377-4886-8f7f-b345bc5e1122 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:10.032824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe24e93-2377-4886-8f7f-b345bc5e1122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:10.033092 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe24e93-2377-4886-8f7f-b345bc5e1122 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:10.033889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe24e93-2377-4886-8f7f-b345bc5e1122 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:10.035666 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe24e93-2377-4886-8f7f-b345bc5e1122 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:10.047096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1430/2834] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:10.051684 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:10.064932 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fb1b7e-4000-4a34-97fa-fcf6c3d503ab admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:10.071515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:10.072411 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:10.072798 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:10.073151 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] PATH_INFO: `/v3/projects/9434ecc5f70a46afb0245019fc0131f6/users/349296d32c4546f3b06c77362d42f78c/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:10.089977 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9434ecc5f70a46afb0245019fc0131f6, user_id=349296d32c4546f3b06c77362d42f78c, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:10.093265 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:10.108929 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1405/2835] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:10 2022] PUT /identity/v3/projects/9434ecc5f70a46afb0245019fc0131f6/users/349296d32c4546f3b06c77362d42f78c/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:10.114055 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1646167e-768a-43e2-9b0e-fe39fd8a61d4 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:10.114338 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1646167e-768a-43e2-9b0e-fe39fd8a61d4 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:10.114628 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1646167e-768a-43e2-9b0e-fe39fd8a61d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:10.137171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1646167e-768a-43e2-9b0e-fe39fd8a61d4 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:10.138387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1646167e-768a-43e2-9b0e-fe39fd8a61d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:10.208341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1431/2836] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:10.218481 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:10.220576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:10.242222 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:10.300057 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253b4c14-bed4-474b-8deb-87556da7deb6 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:10.305321 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:10.305937 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:10.306214 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:10.306472 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:10.316468 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:10.318569 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:10.328568 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1406/2837] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:11.016480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca7615-7570-45ee-af09-2e04df15cdf2 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:11.016958 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca7615-7570-45ee-af09-2e04df15cdf2 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:11.017319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca7615-7570-45ee-af09-2e04df15cdf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:11.043877 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:11.047122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-a2ca7615-7570-45ee-af09-2e04df15cdf2 None None] MFA Rules not processed for user `349296d32c4546f3b06c77362d42f78c`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:11.049321 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a2ca7615-7570-45ee-af09-2e04df15cdf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:11.119534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1432/2838] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:11 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:11.124313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:11.134460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6532c3-29a9-43ba-8912-f7b61a485e5d service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:11.139628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:11.140248 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:11.140487 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:11.140782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] PATH_INFO: `/v3/projects` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:11.141479 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:11.143381 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:11.176826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1407/2839] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:11 2022] POST /identity/v3/projects => generated 405 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:11.177987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca7615-7570-45ee-af09-2e04df15cdf2 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:11.189378 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca7615-7570-45ee-af09-2e04df15cdf2 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:11.247217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:11.247820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:11.248093 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:11.248354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] PATH_INFO: `/v3/users` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:11.248973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:11.251209 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:11.274717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1433/2840] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:11 2022] POST /identity/v3/users => generated 309 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:11.278814 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:11.289348 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef84e29-c379-48e5-98c8-e702cd9f5e36 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:11.294428 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:11.295024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:11.295299 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:11.295565 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:11.296216 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:11.297852 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:11.311780 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1408/2841] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:11.317426 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:11.328078 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ff4e79-1744-40c8-9e50-5cd506227644 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:11.333410 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:11.334090 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:11.334375 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:11.334914 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] PATH_INFO: `/v3/projects/a40f0003d3a84f8e89a978409623ef98/users/ace64b77a8f54d47b5dd481eebd1eac0/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:11.350408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a40f0003d3a84f8e89a978409623ef98, user_id=ace64b77a8f54d47b5dd481eebd1eac0, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:11.353169 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:11.371613 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1434/2842] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:11 2022] PUT /identity/v3/projects/a40f0003d3a84f8e89a978409623ef98/users/ace64b77a8f54d47b5dd481eebd1eac0/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:42:11.376725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:11.387328 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41edf963-af06-4e66-b11d-13c0e76185c3 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:11.460629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd8fe84-cf72-4d3a-b301-e39d8d3e78d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:11.461266 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd8fe84-cf72-4d3a-b301-e39d8d3e78d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:11.461544 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd8fe84-cf72-4d3a-b301-e39d8d3e78d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:11.461805 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd8fe84-cf72-4d3a-b301-e39d8d3e78d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:11.462444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd8fe84-cf72-4d3a-b301-e39d8d3e78d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:11.464079 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd8fe84-cf72-4d3a-b301-e39d8d3e78d7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:11.473939 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1409/2843] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:11.479071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:11.490053 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8f8e29-27cc-4147-8c82-863c6ce2ae18 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:11.495483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:11.496130 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:11.496408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:11.496676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] PATH_INFO: `/v3/projects/a40f0003d3a84f8e89a978409623ef98/users/ace64b77a8f54d47b5dd481eebd1eac0/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:11.498796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a40f0003d3a84f8e89a978409623ef98, user_id=ace64b77a8f54d47b5dd481eebd1eac0, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:11.501460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:11.512123 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1435/2844] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:11 2022] PUT /identity/v3/projects/a40f0003d3a84f8e89a978409623ef98/users/ace64b77a8f54d47b5dd481eebd1eac0/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:12.239989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857e062-c977-430a-bdea-02b0371329e2 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:12.240614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857e062-c977-430a-bdea-02b0371329e2 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:12.240614 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857e062-c977-430a-bdea-02b0371329e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:12.272354 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-f857e062-c977-430a-bdea-02b0371329e2 None None] MFA Rules not processed for user `ace64b77a8f54d47b5dd481eebd1eac0`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:12.273830 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-f857e062-c977-430a-bdea-02b0371329e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:12.350527 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:12.364187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1410/2845] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:12 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:13.501683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:13.503549 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:13.565193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857e062-c977-430a-bdea-02b0371329e2 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:13.571783 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:13.588989 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195752af-65ff-4d2a-b7c6-bea489ee5a7e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:13.594805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:13.595423 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:13.595698 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:13.595967 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:13.597805 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:13.610955 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:13.612615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:13.623330 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1436/2846] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:13.641587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857e062-c977-430a-bdea-02b0371329e2 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:13.647221 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:13.648013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:13.648290 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:13.648557 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] PATH_INFO: `/v3/users/349296d32c4546f3b06c77362d42f78c` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:13.653969 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=349296d32c4546f3b06c77362d42f78c)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:13.655894 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:13.710810 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '349296d32c4546f3b06c77362d42f78c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:13.718364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '349296d32c4546f3b06c77362d42f78c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:13.727793 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '349296d32c4546f3b06c77362d42f78c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:13.728518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '349296d32c4546f3b06c77362d42f78c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:13.729489 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '349296d32c4546f3b06c77362d42f78c'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:13.750724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1411/2847] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:13 2022] DELETE /identity/v3/users/349296d32c4546f3b06c77362d42f78c => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:13.953007 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:13.964573 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7555b838-b12b-4f27-a541-b8ce05f0841f service placement] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:14.026994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:14.027615 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:14.027985 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:14.028168 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] PATH_INFO: `/v3/projects/9434ecc5f70a46afb0245019fc0131f6` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:14.029422 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9434ecc5f70a46afb0245019fc0131f6)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:14.031308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:14.087833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9434ecc5f70a46afb0245019fc0131f6', 'request_id': 'req-a86a1563-195f-4501-b84d-c9a1cfdd7659'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.094020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9434ecc5f70a46afb0245019fc0131f6', 'request_id': 'req-a86a1563-195f-4501-b84d-c9a1cfdd7659'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.099428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9434ecc5f70a46afb0245019fc0131f6', 'request_id': 'req-a86a1563-195f-4501-b84d-c9a1cfdd7659'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.118120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] The token cache is being invalidate because project 9434ecc5f70a46afb0245019fc0131f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:14.118408 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.119104 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.120677 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1437/2848] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:13 2022] DELETE /identity/v3/projects/9434ecc5f70a46afb0245019fc0131f6 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:42:14.124785 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:14.126236 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:14.139999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52701a39-b19f-472f-b4b9-34866cd1ce0d admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:14.216356 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:14.216910 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:14.217217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:14.217486 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] PATH_INFO: `/v3/users/ace64b77a8f54d47b5dd481eebd1eac0` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:14.218749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ace64b77a8f54d47b5dd481eebd1eac0)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:14.220678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:14.235770 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:14.273873 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ace64b77a8f54d47b5dd481eebd1eac0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.280911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ace64b77a8f54d47b5dd481eebd1eac0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.289463 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ace64b77a8f54d47b5dd481eebd1eac0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.290090 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ace64b77a8f54d47b5dd481eebd1eac0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.290708 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-b3e65cc9-ad3f-4960-b83b-715560157e8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ace64b77a8f54d47b5dd481eebd1eac0'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.308902 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1412/2849] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:14 2022] DELETE /identity/v3/users/ace64b77a8f54d47b5dd481eebd1eac0 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:14.507807 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:14.509282 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:14.522602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a1563-195f-4501-b84d-c9a1cfdd7659 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:14.601896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:14.602535 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:14.602825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:14.603191 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] PATH_INFO: `/v3/projects/a40f0003d3a84f8e89a978409623ef98` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:14.604442 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a40f0003d3a84f8e89a978409623ef98)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:14.606683 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:14.625364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:14.659851 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a40f0003d3a84f8e89a978409623ef98', 'request_id': 'req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.667892 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a40f0003d3a84f8e89a978409623ef98', 'request_id': 'req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.674192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a40f0003d3a84f8e89a978409623ef98', 'request_id': 'req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.694664 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] The token cache is being invalidate because project a40f0003d3a84f8e89a978409623ef98 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:14.694948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.695568 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-d7e2aaa3-d2c8-427d-b16b-5a4a0bfe2d3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:14.697144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1438/2850] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:14 2022] DELETE /identity/v3/projects/a40f0003d3a84f8e89a978409623ef98 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:14.703992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e83fb4-3028-40ce-828e-c4c614e3d4dd None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:14.704537 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e83fb4-3028-40ce-828e-c4c614e3d4dd None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:14.704700 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e83fb4-3028-40ce-828e-c4c614e3d4dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:14.727576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-d2e83fb4-3028-40ce-828e-c4c614e3d4dd None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:14.728804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-d2e83fb4-3028-40ce-828e-c4c614e3d4dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:14.801465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1413/2851] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:14.809503 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f0c27-35a1-4447-be1c-29ebb5677f2a None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:14.809781 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f0c27-35a1-4447-be1c-29ebb5677f2a None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:14.810050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f0c27-35a1-4447-be1c-29ebb5677f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:14.830861 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1b6f0c27-35a1-4447-be1c-29ebb5677f2a None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:14.832047 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1b6f0c27-35a1-4447-be1c-29ebb5677f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:14.845022 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1439/2852] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:14.851804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e83fb4-3028-40ce-828e-c4c614e3d4dd None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:14.867364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e83fb4-3028-40ce-828e-c4c614e3d4dd None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:14.873088 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:14.873719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:14.873995 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:14.874261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:14.874926 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:42:14.875263 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:14.876866 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:14.889445 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1414/2853] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:42:14.893940 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6f0c27-35a1-4447-be1c-29ebb5677f2a None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:14.903673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6f0c27-35a1-4447-be1c-29ebb5677f2a None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:14.909465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:14.910052 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:14.910342 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:14.910606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:14.911236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:14.913147 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:14.945020 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1440/2854] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:14 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:14.949331 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:14.960002 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff316667-99d6-4f54-b2d2-b30917d568c0 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.020409 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.021043 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.021341 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.021604 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.022819 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.024721 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.048459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1415/2855] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:14 2022] POST /identity/v3/users => generated 290 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:15.052840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.068572 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c6e18e-8e96-4143-848d-cce1b0296eaa admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.074448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.075472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.075472 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.075707 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.076196 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.077978 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.089679 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1441/2856] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:15 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:15.093761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.103967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc9470-896f-44a3-82d7-5c68ba88cc34 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.108993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4489fa4-14b6-4eb6-ab28-d7663105e88f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.112452 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4489fa4-14b6-4eb6-ab28-d7663105e88f admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.112722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4489fa4-14b6-4eb6-ab28-d7663105e88f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.112989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4489fa4-14b6-4eb6-ab28-d7663105e88f admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.113683 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4489fa4-14b6-4eb6-ab28-d7663105e88f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.115271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4489fa4-14b6-4eb6-ab28-d7663105e88f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.118900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:15.131832 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1416/2857] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:15 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:15.136856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.149652 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5ec214-74df-4c57-89bf-3f98bbcf1fa4 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.155285 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.155930 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.156205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.156473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] PATH_INFO: `/v3/projects/5772726db55644a7817cc9de255603d4/users/ba2920a3fbc24fa8b4dac536a2af525d/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.172223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5772726db55644a7817cc9de255603d4, user_id=ba2920a3fbc24fa8b4dac536a2af525d, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.174916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.191138 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1442/2858] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:15 2022] PUT /identity/v3/projects/5772726db55644a7817cc9de255603d4/users/ba2920a3fbc24fa8b4dac536a2af525d/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:15.263231 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbca60-19c4-4d10-8d87-66e0022a906b None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.263495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbca60-19c4-4d10-8d87-66e0022a906b None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.263736 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbca60-19c4-4d10-8d87-66e0022a906b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.299724 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-bacbca60-19c4-4d10-8d87-66e0022a906b None None] MFA Rules not processed for user `ba2920a3fbc24fa8b4dac536a2af525d`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:15.300943 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-bacbca60-19c4-4d10-8d87-66e0022a906b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:15.372702 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1417/2859] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:15 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:15.377493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.443417 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25d277-4ce4-47b2-870b-d64214589d62 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.449190 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.449799 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.450070 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.450334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.450962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.452843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.483058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1443/2860] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:15 2022] POST /identity/v3/projects => generated 381 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:15.487457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacbca60-19c4-4d10-8d87-66e0022a906b None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.499436 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacbca60-19c4-4d10-8d87-66e0022a906b None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.566980 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.567483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.567756 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.568024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.568703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.570710 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.595994 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1418/2861] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:15 2022] POST /identity/v3/users => generated 297 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:15.601905 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.613510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1ca576-847f-4ba1-9a5b-7b60ac8e6daa admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.618323 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.618926 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.619208 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.619478 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.620120 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.621762 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.632586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1444/2862] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:42:15.637105 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.648268 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a957810-1765-4a5c-b5bd-7e763d20f9f6 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.653645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.654293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.654566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.654837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] PATH_INFO: `/v3/projects/b733a01f9c2d48ec9df17e01f53ec811/users/1e0e5ff7ced848a1ae408763d83b82f6/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.670501 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b733a01f9c2d48ec9df17e01f53ec811, user_id=1e0e5ff7ced848a1ae408763d83b82f6, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.673148 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.694215 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1419/2863] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:15 2022] PUT /identity/v3/projects/b733a01f9c2d48ec9df17e01f53ec811/users/1e0e5ff7ced848a1ae408763d83b82f6/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:42:15.700230 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.719369 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ab0a98-b820-46f0-aee5-6987daaf3216 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.734635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:15.795887 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb764df-c37c-48af-997d-c7288879f762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.796834 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb764df-c37c-48af-997d-c7288879f762 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.797278 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb764df-c37c-48af-997d-c7288879f762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.797713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb764df-c37c-48af-997d-c7288879f762 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.798722 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb764df-c37c-48af-997d-c7288879f762 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.801239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb764df-c37c-48af-997d-c7288879f762 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.813058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1445/2864] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:15 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:15.817718 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.829351 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5acf0b-2c95-4b91-b2cd-e9f2395b1f30 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:15.835332 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:15.835967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.836232 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.836494 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] PATH_INFO: `/v3/projects/b733a01f9c2d48ec9df17e01f53ec811/users/1e0e5ff7ced848a1ae408763d83b82f6/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.838547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b733a01f9c2d48ec9df17e01f53ec811, user_id=1e0e5ff7ced848a1ae408763d83b82f6, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:15.841170 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:15.851938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1420/2865] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:15 2022] PUT /identity/v3/projects/b733a01f9c2d48ec9df17e01f53ec811/users/1e0e5ff7ced848a1ae408763d83b82f6/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:15.862333 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e83c48-3151-41bb-8e2e-58d39c66b54d None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:15.862620 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e83c48-3151-41bb-8e2e-58d39c66b54d None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:15.862888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e83c48-3151-41bb-8e2e-58d39c66b54d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:15.892205 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-28e83c48-3151-41bb-8e2e-58d39c66b54d None None] MFA Rules not processed for user `1e0e5ff7ced848a1ae408763d83b82f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:15.893457 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-28e83c48-3151-41bb-8e2e-58d39c66b54d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:15.976856 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1446/2866] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:15 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:15.991193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:15.992949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:16.038861 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:16.075629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42022e2e-0270-448e-b4b3-5948fdd7f460 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:16.079274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:16.079871 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:16.080143 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:16.080403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:16.090963 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:16.092387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:16.102625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1421/2867] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:15 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:16.186991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e83c48-3151-41bb-8e2e-58d39c66b54d None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:16.199557 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e83c48-3151-41bb-8e2e-58d39c66b54d None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:16.204629 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7974a128-95cb-43f9-adfb-75253d5d7bbe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:16.205283 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7974a128-95cb-43f9-adfb-75253d5d7bbe service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:16.205534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7974a128-95cb-43f9-adfb-75253d5d7bbe service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:16.205820 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7974a128-95cb-43f9-adfb-75253d5d7bbe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:16.267227 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7974a128-95cb-43f9-adfb-75253d5d7bbe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:16.268704 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7974a128-95cb-43f9-adfb-75253d5d7bbe service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:16.278235 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1447/2868] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:16 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:26.097660 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=17 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:26.273718 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=84 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:27.436112 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:27.503783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9e2cdd-96c0-4652-8afa-aa5c22e8a6a5 service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:27.508917 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:27.509577 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:27.509854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:27.510121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] PATH_INFO: `/v3/users/ba2920a3fbc24fa8b4dac536a2af525d` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:27.511315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba2920a3fbc24fa8b4dac536a2af525d)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:27.513260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:27.567942 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba2920a3fbc24fa8b4dac536a2af525d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:27.574911 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba2920a3fbc24fa8b4dac536a2af525d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:27.583599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba2920a3fbc24fa8b4dac536a2af525d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:27.585060 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba2920a3fbc24fa8b4dac536a2af525d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:27.585749 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba2920a3fbc24fa8b4dac536a2af525d'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:27.609400 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1422/2869] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:27 2022] DELETE /identity/v3/users/ba2920a3fbc24fa8b4dac536a2af525d => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:27.615345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:27.615863 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:27.615968 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:27.635865 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:27.637071 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:27.712193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1448/2870] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:27.724524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:27.726927 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:27.794562 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9024ab77-8313-4a64-856b-60dfdde3e193 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:27.801473 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:27.802431 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:27.803055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:27.803055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:27.814837 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:27.816271 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:27.827042 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1423/2871] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:28.128962 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:28.131019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:28.143965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fdbec0-a491-4ca6-ab78-cfaf8c87f309 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:28.151002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:28.151654 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:28.151932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:28.152213 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] PATH_INFO: `/v3/projects/5772726db55644a7817cc9de255603d4` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:28.153479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5772726db55644a7817cc9de255603d4)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:28.155424 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:28.206569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5772726db55644a7817cc9de255603d4', 'request_id': 'req-f3032486-12b8-436b-96c5-69fb4c12f68e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.213350 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5772726db55644a7817cc9de255603d4', 'request_id': 'req-f3032486-12b8-436b-96c5-69fb4c12f68e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.220028 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5772726db55644a7817cc9de255603d4', 'request_id': 'req-f3032486-12b8-436b-96c5-69fb4c12f68e'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.234466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] The token cache is being invalidate because project 5772726db55644a7817cc9de255603d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:28.234746 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.235363 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.236875 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1449/2872] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:28 2022] DELETE /identity/v3/projects/5772726db55644a7817cc9de255603d4 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:28.241276 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:28.252401 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e1c2e0-583f-4158-8b32-c3f2eab1c6b7 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:28.272258 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:28.320694 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:28.321288 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:28.321561 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:28.321824 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] PATH_INFO: `/v3/users/1e0e5ff7ced848a1ae408763d83b82f6` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:28.322993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e0e5ff7ced848a1ae408763d83b82f6)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:28.324893 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:28.379449 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e0e5ff7ced848a1ae408763d83b82f6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.387781 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e0e5ff7ced848a1ae408763d83b82f6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.398940 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e0e5ff7ced848a1ae408763d83b82f6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.399613 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e0e5ff7ced848a1ae408763d83b82f6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.400524 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-87ccf844-e827-4978-8919-d23e1f58c114 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e0e5ff7ced848a1ae408763d83b82f6'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.426867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1424/2873] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:28 2022] DELETE /identity/v3/users/1e0e5ff7ced848a1ae408763d83b82f6 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:28.748358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:28.750180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:28.763836 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3032486-12b8-436b-96c5-69fb4c12f68e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:28.828481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:28.829112 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:28.829414 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:28.829681 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] PATH_INFO: `/v3/projects/b733a01f9c2d48ec9df17e01f53ec811` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:28.831043 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b733a01f9c2d48ec9df17e01f53ec811)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:28.832723 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:28.871197 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:28.888393 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b733a01f9c2d48ec9df17e01f53ec811', 'request_id': 'req-bb3d7605-c28a-4835-968c-f8784564b3a3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.898042 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b733a01f9c2d48ec9df17e01f53ec811', 'request_id': 'req-bb3d7605-c28a-4835-968c-f8784564b3a3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.905223 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b733a01f9c2d48ec9df17e01f53ec811', 'request_id': 'req-bb3d7605-c28a-4835-968c-f8784564b3a3'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.919248 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] The token cache is being invalidate because project b733a01f9c2d48ec9df17e01f53ec811 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:28.919667 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.920631 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-bb3d7605-c28a-4835-968c-f8784564b3a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:28.922994 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1450/2874] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:28 2022] DELETE /identity/v3/projects/b733a01f9c2d48ec9df17e01f53ec811 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:28.930513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa5dbe8-d414-4dc2-afa1-4ac90da0918f None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:28.931193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa5dbe8-d414-4dc2-afa1-4ac90da0918f None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:28.931193 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa5dbe8-d414-4dc2-afa1-4ac90da0918f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:28.956422 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-0fa5dbe8-d414-4dc2-afa1-4ac90da0918f None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:28.957689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-0fa5dbe8-d414-4dc2-afa1-4ac90da0918f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:29.020499 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1425/2875] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:29.030534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6882676b-6e4d-46dd-ba39-a284960c8098 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.030825 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6882676b-6e4d-46dd-ba39-a284960c8098 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.031101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6882676b-6e4d-46dd-ba39-a284960c8098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.053506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-6882676b-6e4d-46dd-ba39-a284960c8098 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:29.054719 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6882676b-6e4d-46dd-ba39-a284960c8098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:29.065869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1451/2876] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:29.070310 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa5dbe8-d414-4dc2-afa1-4ac90da0918f None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.081990 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa5dbe8-d414-4dc2-afa1-4ac90da0918f None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.088583 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.089314 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.089623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.089919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.090698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:42:29.091012 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.093241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.105402 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1426/2877] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:42:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:42:29.110347 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6882676b-6e4d-46dd-ba39-a284960c8098 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.121488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6882676b-6e4d-46dd-ba39-a284960c8098 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.126637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.127238 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.127513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.127778 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.128420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.130382 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.160324 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1452/2878] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:29 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 27 02:42:29.165219 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.179243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525bd58a-ccb9-481b-bb91-f8422661ff31 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.206789 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:29.243417 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.244030 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.244303 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.244584 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.245325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.247256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.272392 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1427/2879] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:29 2022] POST /identity/v3/users => generated 290 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:29.277095 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.288373 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d786a385-f488-4762-abbb-156e42cdbe2c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.294780 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.295144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.295824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.296218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.297303 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.299192 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.311608 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1453/2880] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:29.315629 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.325759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22489fb0-170c-4dd7-8a41-260a9c2932d7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.330803 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cd115d-a844-4ff5-8008-e544c26bf250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.331387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cd115d-a844-4ff5-8008-e544c26bf250 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.331658 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cd115d-a844-4ff5-8008-e544c26bf250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.331951 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cd115d-a844-4ff5-8008-e544c26bf250 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.332593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cd115d-a844-4ff5-8008-e544c26bf250 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.334364 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cd115d-a844-4ff5-8008-e544c26bf250 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.347442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1428/2881] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:29.354341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.368726 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be03937-0857-4a15-a269-09c8931db59e admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.374747 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.375336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.375614 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.375878 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] PATH_INFO: `/v3/projects/265f2162510c46908e1a2f8d0e2c40e0/users/5ae7014cad1a4bf1955f96b29b70592b/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.391179 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=265f2162510c46908e1a2f8d0e2c40e0, user_id=5ae7014cad1a4bf1955f96b29b70592b, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.393840 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.409988 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1454/2882] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:29 2022] PUT /identity/v3/projects/265f2162510c46908e1a2f8d0e2c40e0/users/5ae7014cad1a4bf1955f96b29b70592b/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:29.421021 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f688ca-a266-4b40-9664-4e189d76db87 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.421329 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f688ca-a266-4b40-9664-4e189d76db87 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.421593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f688ca-a266-4b40-9664-4e189d76db87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.452651 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b1f688ca-a266-4b40-9664-4e189d76db87 None None] MFA Rules not processed for user `5ae7014cad1a4bf1955f96b29b70592b`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:29.453971 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b1f688ca-a266-4b40-9664-4e189d76db87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:29.526094 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1429/2883] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:29 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:29.533389 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.605717 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867f24a-f584-4ebc-ab48-17c1ce05fbf4 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.612319 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.612932 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.613263 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.613525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.614607 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.617144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.650606 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1455/2884] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:29 2022] POST /identity/v3/projects => generated 381 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:29.654695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f688ca-a266-4b40-9664-4e189d76db87 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.668423 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f688ca-a266-4b40-9664-4e189d76db87 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.730293 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.730912 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.731185 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.731454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.732110 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.734055 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.760363 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1430/2885] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:29 2022] POST /identity/v3/users => generated 297 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:29.764579 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.776513 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f05fd8-04c7-4af2-8aa7-2e535a1d6885 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.782467 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.783064 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.783311 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.783569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.784218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.785853 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.797592 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1456/2886] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:29.802776 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.814139 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e838bc7-2b47-4235-93c0-eb3015a43742 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.820224 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.820889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.821153 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.821467 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] PATH_INFO: `/v3/projects/b9ad8442755545ea97279084233b6d1f/users/ae45514327404f27bc633a612f6854d2/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.842350 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9ad8442755545ea97279084233b6d1f, user_id=ae45514327404f27bc633a612f6854d2, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.845078 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.861073 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1431/2887] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:29 2022] PUT /identity/v3/projects/b9ad8442755545ea97279084233b6d1f/users/ae45514327404f27bc633a612f6854d2/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:42:29.865511 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.875450 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7922fd2-8393-45a2-beb2-14b66864dc9b admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.895680 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:29.945387 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8975f76-45ea-496a-88a6-2ff94aab03b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.945916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8975f76-45ea-496a-88a6-2ff94aab03b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.946186 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8975f76-45ea-496a-88a6-2ff94aab03b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.946453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8975f76-45ea-496a-88a6-2ff94aab03b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.947091 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8975f76-45ea-496a-88a6-2ff94aab03b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.948673 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8975f76-45ea-496a-88a6-2ff94aab03b7 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:29.959479 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1457/2888] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:29 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:29.966238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:29.973177 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:29.981692 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a63596-0c9f-4440-a70f-ea3eceb13dd1 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:29.987625 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:29.988272 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:29.988545 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:29.988821 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] PATH_INFO: `/v3/projects/b9ad8442755545ea97279084233b6d1f/users/ae45514327404f27bc633a612f6854d2/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:29.991379 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9ad8442755545ea97279084233b6d1f, user_id=ae45514327404f27bc633a612f6854d2, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:29.994059 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:30.005703 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1432/2889] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:29 2022] PUT /identity/v3/projects/b9ad8442755545ea97279084233b6d1f/users/ae45514327404f27bc633a612f6854d2/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:30.016096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5472658a-8a51-43aa-9427-dad66407e6df None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:30.016374 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5472658a-8a51-43aa-9427-dad66407e6df None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:30.016646 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5472658a-8a51-43aa-9427-dad66407e6df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:30.051793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-5472658a-8a51-43aa-9427-dad66407e6df None None] MFA Rules not processed for user `ae45514327404f27bc633a612f6854d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:30.053252 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-5472658a-8a51-43aa-9427-dad66407e6df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:30.125361 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1458/2890] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:30 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:30.139386 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:30.142353 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:30.204182 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1136e3f-123e-45bd-ace2-1419f0b9de04 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:30.208757 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:30.209218 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:30.209495 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:30.209761 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:30.219461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:30.220892 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:30.230711 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1433/2891] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:30 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:30.323939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5472658a-8a51-43aa-9427-dad66407e6df None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:30.336017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5472658a-8a51-43aa-9427-dad66407e6df None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:30.340236 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8165aa7f-1884-4aa2-97f8-cb2aaa36e92e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:30.340832 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165aa7f-1884-4aa2-97f8-cb2aaa36e92e service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:30.341106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165aa7f-1884-4aa2-97f8-cb2aaa36e92e service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:30.341400 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165aa7f-1884-4aa2-97f8-cb2aaa36e92e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:30.404187 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8165aa7f-1884-4aa2-97f8-cb2aaa36e92e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:30.405649 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8165aa7f-1884-4aa2-97f8-cb2aaa36e92e service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:30.414980 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1459/2892] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:30 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:30.739728 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:30.812197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd8fc41-55d4-435c-916e-3f67a1b619cb service cinder] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:30.816968 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:30.817628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:30.817981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:30.818251 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] PATH_INFO: `/v3/users/5ae7014cad1a4bf1955f96b29b70592b` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:30.819406 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ae7014cad1a4bf1955f96b29b70592b)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:30.821378 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:30.873454 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ae7014cad1a4bf1955f96b29b70592b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:30.882196 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ae7014cad1a4bf1955f96b29b70592b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:30.892542 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ae7014cad1a4bf1955f96b29b70592b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:30.893183 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ae7014cad1a4bf1955f96b29b70592b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:30.893797 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ae7014cad1a4bf1955f96b29b70592b'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:30.918615 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1434/2893] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:30 2022] DELETE /identity/v3/users/5ae7014cad1a4bf1955f96b29b70592b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:30.925706 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:30.926460 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:30.926626 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:30.951921 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:30.953113 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:30.996063 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=100 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:31.018874 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1460/2894] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:31.031050 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:31.032572 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:31.047900 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:31.103521 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987153a-d57f-455a-a86c-cde2039ea67c admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:31.107593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:31.108199 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:31.108470 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:31.108738 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:31.120137 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:31.121722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:31.132667 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1435/2895] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:31.426048 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:31.428256 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:31.440900 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1938a404-3a58-4b9d-a112-ae9d8f5dbed3 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:31.446366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:31.447016 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:31.447862 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:31.448249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] PATH_INFO: `/v3/projects/265f2162510c46908e1a2f8d0e2c40e0` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:31.449624 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] RBAC: Authorizing `identity:delete_project(project_id=265f2162510c46908e1a2f8d0e2c40e0)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:31.451642 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:31.505219 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '265f2162510c46908e1a2f8d0e2c40e0', 'request_id': 'req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.512114 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '265f2162510c46908e1a2f8d0e2c40e0', 'request_id': 'req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.517839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '265f2162510c46908e1a2f8d0e2c40e0', 'request_id': 'req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.532206 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] The token cache is being invalidate because project 265f2162510c46908e1a2f8d0e2c40e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:31.532483 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.533081 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.534600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1461/2896] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:31 2022] DELETE /identity/v3/projects/265f2162510c46908e1a2f8d0e2c40e0 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:31.538674 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:31.549313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70616b4d-e6a1-4afd-a74a-79818788f9fe service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:31.610024 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:31.610645 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:31.610915 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:31.611175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] PATH_INFO: `/v3/users/ae45514327404f27bc633a612f6854d2` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:31.612297 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae45514327404f27bc633a612f6854d2)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:31.614227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:31.674589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae45514327404f27bc633a612f6854d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.684206 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae45514327404f27bc633a612f6854d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.696233 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae45514327404f27bc633a612f6854d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.697109 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae45514327404f27bc633a612f6854d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.697922 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-355ca12d-0ff7-42e3-83c1-7020a6312301 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae45514327404f27bc633a612f6854d2'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:31.724442 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1436/2897] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:31 2022] DELETE /identity/v3/users/ae45514327404f27bc633a612f6854d2 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:32.031030 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.032569 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:32.049243 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f793aaa2-8ab4-45e6-8ba8-80e6ae58259c admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.117019 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.117655 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.117965 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.118234 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] PATH_INFO: `/v3/projects/b9ad8442755545ea97279084233b6d1f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.119377 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9ad8442755545ea97279084233b6d1f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.121302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.178790 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9ad8442755545ea97279084233b6d1f', 'request_id': 'req-2104086c-f884-4ea1-9868-45dfdf9dff68'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:32.187344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9ad8442755545ea97279084233b6d1f', 'request_id': 'req-2104086c-f884-4ea1-9868-45dfdf9dff68'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:32.194495 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9ad8442755545ea97279084233b6d1f', 'request_id': 'req-2104086c-f884-4ea1-9868-45dfdf9dff68'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:32.204257 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:32.209551 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] The token cache is being invalidate because project b9ad8442755545ea97279084233b6d1f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:32.210217 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:32.210993 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-2104086c-f884-4ea1-9868-45dfdf9dff68 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:32.212544 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1462/2898] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:32 2022] DELETE /identity/v3/projects/b9ad8442755545ea97279084233b6d1f => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:42:32.217426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b929814d-d863-455f-a4bd-4f04262b351e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.217906 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b929814d-d863-455f-a4bd-4f04262b351e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.218053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b929814d-d863-455f-a4bd-4f04262b351e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.240260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-b929814d-d863-455f-a4bd-4f04262b351e None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:32.241628 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-b929814d-d863-455f-a4bd-4f04262b351e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:32.275187 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:32.317699 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1437/2899] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:32.325995 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5398abe-8e5e-49ed-8947-40d522693def None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.326298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5398abe-8e5e-49ed-8947-40d522693def None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.326577 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5398abe-8e5e-49ed-8947-40d522693def None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.347332 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c5398abe-8e5e-49ed-8947-40d522693def None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:32.348541 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c5398abe-8e5e-49ed-8947-40d522693def None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:32.361401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1463/2900] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:32.366316 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b929814d-d863-455f-a4bd-4f04262b351e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.377347 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b929814d-d863-455f-a4bd-4f04262b351e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.382898 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.383538 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.383813 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.384086 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.384748 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:42:32.385049 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.386677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.399781 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1438/2901] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:42:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:42:32.407210 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5398abe-8e5e-49ed-8947-40d522693def None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.418533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5398abe-8e5e-49ed-8947-40d522693def None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.424509 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.425106 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.425412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.425676 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.426310 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.428510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.468481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1464/2902] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:32 2022] POST /identity/v3/projects => generated 375 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:32.473171 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.484041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ac0091-53c0-4a32-9893-a3fd2e2980cb admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.548388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.549048 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.549359 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.549627 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.550270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.552439 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.575741 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1439/2903] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:32 2022] POST /identity/v3/users => generated 288 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:32.580087 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.591543 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d17365c-814a-4ba3-95a0-af9a7f107232 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.596500 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.597125 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.597425 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.597694 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.598344 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.600129 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.610603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1465/2904] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:32.614870 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.625504 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2fa08b-e2e3-42f9-8f77-8ed018ec8aca admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.630788 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f15f12-7948-449b-82aa-323d90d683a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.631308 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f15f12-7948-449b-82aa-323d90d683a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.631576 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f15f12-7948-449b-82aa-323d90d683a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.631841 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f15f12-7948-449b-82aa-323d90d683a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.632497 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f15f12-7948-449b-82aa-323d90d683a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.634114 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f15f12-7948-449b-82aa-323d90d683a9 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.644730 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1440/2905] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:32.649943 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.662275 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9702e25-40ca-4b1f-bccd-624af4a98cfb admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.668139 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.668831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.669105 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.669384 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] PATH_INFO: `/v3/projects/15fda0215ef645a38f3a8ab3ccca677c/users/779d55686be44998b259c1489d0db552/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.685444 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15fda0215ef645a38f3a8ab3ccca677c, user_id=779d55686be44998b259c1489d0db552, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.688075 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.706589 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1466/2906] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:32 2022] PUT /identity/v3/projects/15fda0215ef645a38f3a8ab3ccca677c/users/779d55686be44998b259c1489d0db552/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:32.717889 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e8595-6c74-457c-a833-a46da710c416 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.718197 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e8595-6c74-457c-a833-a46da710c416 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.718465 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e8595-6c74-457c-a833-a46da710c416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.748815 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-108e8595-6c74-457c-a833-a46da710c416 None None] MFA Rules not processed for user `779d55686be44998b259c1489d0db552`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:32.750061 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-108e8595-6c74-457c-a833-a46da710c416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:32.830653 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1441/2907] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:32 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:32.845659 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:32.847833 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:32.913916 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc738e7-9e61-44e1-90e5-71e11f5d6e51 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:32.917733 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76621821-5ba5-47a9-acda-8f203f3017f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '598bc689672741c0ac600e42cf45e39d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:32.918343 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76621821-5ba5-47a9-acda-8f203f3017f8 service cinder] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:32.918622 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76621821-5ba5-47a9-acda-8f203f3017f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:32.918886 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76621821-5ba5-47a9-acda-8f203f3017f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:32.928465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76621821-5ba5-47a9-acda-8f203f3017f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:32.929937 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76621821-5ba5-47a9-acda-8f203f3017f8 service cinder] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:32.939973 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1467/2908] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:32 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:36.109168 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108e8595-6c74-457c-a833-a46da710c416 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:36.151203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:36.187336 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108e8595-6c74-457c-a833-a46da710c416 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:36.192518 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:36.193175 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:36.193460 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:36.193729 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] PATH_INFO: `/v3/users/779d55686be44998b259c1489d0db552` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:36.194998 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=779d55686be44998b259c1489d0db552)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:36.196934 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:36.254044 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '779d55686be44998b259c1489d0db552'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.263919 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '779d55686be44998b259c1489d0db552'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.273967 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '779d55686be44998b259c1489d0db552'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.274732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '779d55686be44998b259c1489d0db552'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.275457 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '779d55686be44998b259c1489d0db552'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.298669 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1442/2909] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:42:36 2022] DELETE /identity/v3/users/779d55686be44998b259c1489d0db552 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:36.306072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:36.306072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:36.306072 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:36.325358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:36.326824 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:36.394519 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1468/2910] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:36.406243 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:36.407695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:36.479104 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2cde61-18b6-40b1-bea6-6b3f250af2a2 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:36.483828 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb681d1-cdd9-4d18-8fc8-d98af6a248e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:36.484444 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb681d1-cdd9-4d18-8fc8-d98af6a248e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:36.484718 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb681d1-cdd9-4d18-8fc8-d98af6a248e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:36.484987 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb681d1-cdd9-4d18-8fc8-d98af6a248e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:36.498679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb681d1-cdd9-4d18-8fc8-d98af6a248e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:36.500128 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb681d1-cdd9-4d18-8fc8-d98af6a248e5 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:36.511227 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1443/2911] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:36.826839 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:36.828322 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:36.841939 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d1cdcc-db02-41de-9054-1c2a5a446427 None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:36.846725 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:36.847396 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:36.847672 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:36.847946 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] PATH_INFO: `/v3/projects/15fda0215ef645a38f3a8ab3ccca677c` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:36.849328 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=15fda0215ef645a38f3a8ab3ccca677c)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:36.851171 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:36.879108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:36.918435 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15fda0215ef645a38f3a8ab3ccca677c', 'request_id': 'req-943e6293-9c2b-4480-bb93-2c22b599ee5a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.924998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15fda0215ef645a38f3a8ab3ccca677c', 'request_id': 'req-943e6293-9c2b-4480-bb93-2c22b599ee5a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.931176 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15fda0215ef645a38f3a8ab3ccca677c', 'request_id': 'req-943e6293-9c2b-4480-bb93-2c22b599ee5a'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.946366 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] The token cache is being invalidate because project 15fda0215ef645a38f3a8ab3ccca677c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:42:36.946596 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.947298 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-943e6293-9c2b-4480-bb93-2c22b599ee5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:42:36.948782 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1469/2912] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:42:36 2022] DELETE /identity/v3/projects/15fda0215ef645a38f3a8ab3ccca677c => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:36.954051 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c8b4a-12d7-48ee-84d8-c7c3aeadfadb None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:36.954399 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c8b4a-12d7-48ee-84d8-c7c3aeadfadb None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:36.954744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c8b4a-12d7-48ee-84d8-c7c3aeadfadb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:36.980198 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-052c8b4a-12d7-48ee-84d8-c7c3aeadfadb None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:36.981459 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-052c8b4a-12d7-48ee-84d8-c7c3aeadfadb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:37.030678 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:37.048882 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1444/2913] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:37.058294 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26677bc8-5b40-44c0-9865-09b96b52f41c None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.058555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26677bc8-5b40-44c0-9865-09b96b52f41c None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.058869 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26677bc8-5b40-44c0-9865-09b96b52f41c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.085158 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-26677bc8-5b40-44c0-9865-09b96b52f41c None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:37.086534 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-26677bc8-5b40-44c0-9865-09b96b52f41c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:37.096552 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1470/2914] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:37.102267 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c8b4a-12d7-48ee-84d8-c7c3aeadfadb None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:37.113261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c8b4a-12d7-48ee-84d8-c7c3aeadfadb None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:37.119085 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:37.120854 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.121172 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.121455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] PATH_INFO: `/v3/domains` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.122123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81746) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 27 02:42:37.122427 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:37.124025 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:37.136609 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1445/2915] 10.222.0.52 () {58 vars in 1153 bytes} [Sat Aug 27 02:42:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 27 02:42:37.143122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26677bc8-5b40-44c0-9865-09b96b52f41c None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:37.153611 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26677bc8-5b40-44c0-9865-09b96b52f41c None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:37.159635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:37.160237 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.160510 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.160812 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.161947 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:37.164466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:37.204758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1471/2916] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:37 2022] POST /identity/v3/projects => generated 385 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:37.209426 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:37.219560 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be73712a-7e32-471e-bc0a-f64de5e14820 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:37.277531 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:37.278123 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.278410 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.278679 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.279412 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:37.281261 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:37.306387 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1446/2917] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:37 2022] POST /identity/v3/users => generated 293 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:37.310888 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:37.322354 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e84a5a-bd59-4a12-876b-fb42bf9df947 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:37.328041 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:37.328696 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.328976 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.329336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.329930 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:37.331635 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:37.344357 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1472/2918] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:37 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:37.349403 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:37.359731 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e93cc55-6809-440c-99e2-8e4a143d4f2f admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:37.365116 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3885136-5f3b-4edc-90ff-1ff7dd397f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:37.365783 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3885136-5f3b-4edc-90ff-1ff7dd397f71 admin admin] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.366083 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3885136-5f3b-4edc-90ff-1ff7dd397f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.366343 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3885136-5f3b-4edc-90ff-1ff7dd397f71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.367007 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3885136-5f3b-4edc-90ff-1ff7dd397f71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:37.368663 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3885136-5f3b-4edc-90ff-1ff7dd397f71 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:37.380589 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1447/2919] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 27 02:42:37.385267 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:37.395975 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd8970f-aa49-400d-8d33-763d232dbeca admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:37.401101 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:37.401796 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] REQUEST_METHOD: `PUT` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.402069 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.402341 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] PATH_INFO: `/v3/projects/7dd5f20c4a714ee3b555d4aef8b87730/users/ba73716667184588b522bc7856404be8/roles/51f5bbe85ea14c879a8c35831e619357` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.419108 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dd5f20c4a714ee3b555d4aef8b87730, user_id=ba73716667184588b522bc7856404be8, role_id=51f5bbe85ea14c879a8c35831e619357)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:37.421793 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:37.435908 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1473/2920] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:37 2022] PUT /identity/v3/projects/7dd5f20c4a714ee3b555d4aef8b87730/users/ba73716667184588b522bc7856404be8/roles/51f5bbe85ea14c879a8c35831e619357 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:37.441313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb1fc42-bf8c-4fd7-b180-a66ae5028c14 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.441592 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb1fc42-bf8c-4fd7-b180-a66ae5028c14 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.441858 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb1fc42-bf8c-4fd7-b180-a66ae5028c14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.461689 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-fcb1fc42-bf8c-4fd7-b180-a66ae5028c14 None None] MFA Rules not processed for user `6648709a4b6d470c80e9b42cb0d345ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:37.462808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fcb1fc42-bf8c-4fd7-b180-a66ae5028c14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:37.528201 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1448/2921] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 27 02:42:37.538308 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:37.540017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:37.613784 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb48c6-ca30-4c48-b746-cf2184147794 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:37.618221 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:37.618838 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:37.619098 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:37.619364 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:37.629405 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:37.631004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:37.641302 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1474/2922] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:42.349623 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f26c2b0-ee87-4a95-a103-afd2b3c88390 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:42.350292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f26c2b0-ee87-4a95-a103-afd2b3c88390 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:42.350292 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f26c2b0-ee87-4a95-a103-afd2b3c88390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:42.394342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-5f26c2b0-ee87-4a95-a103-afd2b3c88390 None None] MFA Rules not processed for user `ba73716667184588b522bc7856404be8`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:42.395779 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-5f26c2b0-ee87-4a95-a103-afd2b3c88390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:42.451046 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:42.481162 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1449/2923] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:42.485842 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:42.498829 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3aeffb-d7ac-41cb-8975-19873de6c77e service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:42.504920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:42.506184 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:42.506758 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:42.507239 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:42.508381 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:42.511099 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:42.544948 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1475/2924] 10.222.0.52 () {60 vars in 1153 bytes} [Sat Aug 27 02:42:42 2022] POST /identity/v3/projects => generated 385 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:42.549666 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f26c2b0-ee87-4a95-a103-afd2b3c88390 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:42.563579 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f26c2b0-ee87-4a95-a103-afd2b3c88390 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:42.644551 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:42.645944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:42.645944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:42.646238 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] PATH_INFO: `/v3/users` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:42.647077 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:42.649053 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:42.677062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1450/2925] 10.222.0.52 () {60 vars in 1144 bytes} [Sat Aug 27 02:42:42 2022] POST /identity/v3/users => generated 299 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 27 02:42:42.681180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:42.695831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a724209-1315-4fb9-97ee-2438a9704fc2 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:42.700788 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:42.701448 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:42.701729 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:42.702002 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:42.702658 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:42.704262 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:42.723453 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1476/2926] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:42 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:42.727626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:42.739313 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2d0c3d-0803-4615-9734-63e50a6de2fc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:42.744150 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:42.744850 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:42.745337 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:42.745752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] PATH_INFO: `/v3/projects/71573f3978164a2894f5b9632e0722de/users/eab0409bbbbb4b39a0fc34421cfdd377/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:42.763509 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=71573f3978164a2894f5b9632e0722de, user_id=eab0409bbbbb4b39a0fc34421cfdd377, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:42.766391 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:42.788301 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1451/2927] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:42 2022] PUT /identity/v3/projects/71573f3978164a2894f5b9632e0722de/users/eab0409bbbbb4b39a0fc34421cfdd377/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 27 02:42:42.793465 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:42.810831 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1b8599-0447-4a47-b7a0-1beaa1ae6bc9 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:42.835336 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:42.897481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1222c196-3076-4edc-ba35-dea22a2d7068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:42.898174 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222c196-3076-4edc-ba35-dea22a2d7068 admin admin] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:42.898466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222c196-3076-4edc-ba35-dea22a2d7068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:42.899153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222c196-3076-4edc-ba35-dea22a2d7068 admin admin] PATH_INFO: `/v3/roles` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:42.899368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1222c196-3076-4edc-ba35-dea22a2d7068 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:42.901074 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1222c196-3076-4edc-ba35-dea22a2d7068 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:42.912991 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1477/2928] 10.222.0.52 () {58 vars in 1122 bytes} [Sat Aug 27 02:42:42 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 27 02:42:42.917405 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:42.928773 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee4daa5-29b0-417c-b4e7-8a10b29929fb admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:42.933938 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:42.934978 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:42.935274 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:42.935554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] PATH_INFO: `/v3/projects/71573f3978164a2894f5b9632e0722de/users/eab0409bbbbb4b39a0fc34421cfdd377/roles/b83e461b8f6643fba45f427d28498c41` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:42.937637 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71573f3978164a2894f5b9632e0722de, user_id=eab0409bbbbb4b39a0fc34421cfdd377, role_id=b83e461b8f6643fba45f427d28498c41)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:42.940270 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:42.953782 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1452/2929] 10.222.0.52 () {60 vars in 1483 bytes} [Sat Aug 27 02:42:42 2022] PUT /identity/v3/projects/71573f3978164a2894f5b9632e0722de/users/eab0409bbbbb4b39a0fc34421cfdd377/roles/b83e461b8f6643fba45f427d28498c41 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:42:47.402637 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0efc0f-4090-4251-a594-b55d49569b42 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:47.403180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0efc0f-4090-4251-a594-b55d49569b42 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:47.403180 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0efc0f-4090-4251-a594-b55d49569b42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:47.436789 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-3f0efc0f-4090-4251-a594-b55d49569b42 None None] MFA Rules not processed for user `eab0409bbbbb4b39a0fc34421cfdd377`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:47.438050 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3f0efc0f-4090-4251-a594-b55d49569b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:47.522773 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1478/2930] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:47 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:47.534958 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:47.537820 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:47.640676 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa288a0-39b7-49f7-a7a5-2c3dc55c95d7 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:47.644448 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab71b242a16641429c29dc8aa67867e5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:47.645420 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:47.645695 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:47.645981 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:47.681500 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:47.707877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:47.709759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:47.720241 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1453/2931] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:42:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:42:47.857397 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f5bc9-4073-43f1-821e-1d668d058658 None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:47.857785 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f5bc9-4073-43f1-821e-1d668d058658 None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:47.858156 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f5bc9-4073-43f1-821e-1d668d058658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:47.880899 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-bb0f5bc9-4073-43f1-821e-1d668d058658 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:47.882533 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-bb0f5bc9-4073-43f1-821e-1d668d058658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:48.080193 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1479/2932] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:42:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:48.133203 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:48.135771 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:48.221580 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d83526e-92d0-4f20-a6d4-c939d40621c2 service glance] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:48.227707 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4249fe1a-7182-48aa-887b-0c74191e98c1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:48.228315 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4249fe1a-7182-48aa-887b-0c74191e98c1 service swift] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:48.228587 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4249fe1a-7182-48aa-887b-0c74191e98c1 service swift] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:48.228853 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4249fe1a-7182-48aa-887b-0c74191e98c1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:48.237746 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4249fe1a-7182-48aa-887b-0c74191e98c1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:48.239857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4249fe1a-7182-48aa-887b-0c74191e98c1 service swift] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:48.248342 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1454/2933] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:42:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:42:56.889349 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0437a-ef6d-4797-a239-361442bc0c4b None None] REQUEST_METHOD: `POST` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:56.889901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0437a-ef6d-4797-a239-361442bc0c4b None None] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:56.889901 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0437a-ef6d-4797-a239-361442bc0c4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:56.911504 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.auth.core [None req-c0d0437a-ef6d-4797-a239-361442bc0c4b None None] MFA Rules not processed for user `ba73716667184588b522bc7856404be8`. Rule list: `[]` (Enabled: `True`). {{(pid=81745) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:56.912690 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c0d0437a-ef6d-4797-a239-361442bc0c4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:56.919734 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1480/2934] 10.222.0.52 () {58 vars in 958 bytes} [Sat Aug 27 02:42:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:58.244831 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=44 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:42:59.115961 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8454ba-8614-4260-b9ad-1ec9a6894320 None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:59.116228 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8454ba-8614-4260-b9ad-1ec9a6894320 None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:59.116492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8454ba-8614-4260-b9ad-1ec9a6894320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:59.136918 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-8f8454ba-8614-4260-b9ad-1ec9a6894320 None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:42:59.138039 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8f8454ba-8614-4260-b9ad-1ec9a6894320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:42:59.145482 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1455/2935] 10.222.0.52 () {58 vars in 1009 bytes} [Sat Aug 27 02:42:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:42:59.161661 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0437a-ef6d-4797-a239-361442bc0c4b None None] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:42:59.175301 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0437a-ef6d-4797-a239-361442bc0c4b None None] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:42:59.182144 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:42:59.182760 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:42:59.183034 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:42:59.183430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:42:59.193004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:42:59.194713 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:42:59.203076 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1481/2936] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:42:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:43:09.131751 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=3 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:43:09.199843 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:43:29.309103 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8454ba-8614-4260-b9ad-1ec9a6894320 None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:43:29.373217 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8454ba-8614-4260-b9ad-1ec9a6894320 None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:43:29.377722 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:43:29.378333 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:43:29.378599 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:43:29.378857 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:43:29.388344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:43:29.389804 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:43:29.399983 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1456/2937] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:43:29 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:43:31.429555 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:43:31.432046 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:43:31.554602 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9850757-ef2c-4a1e-b83c-30d457d248e7 service swift] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:43:31.564018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c83f0d0e2e8047a98fbe114ceb15c6e0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:43:31.564018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:43:31.564018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:43:31.564018 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:43:31.578395 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:43:31.599288 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:43:31.600889 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:43:31.613017 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1482/2938] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:43:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:43:39.100636 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:43:39.112744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8291d8-1427-404e-b1c8-42996f62a72e service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:43:39.117475 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:43:39.118455 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:43:39.118725 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:43:39.118992 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:43:39.120944 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:43:39.132867 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:43:39.134319 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:43:39.143360 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1457/2939] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:43:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:43:41.607220 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=43 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:43:49.138682 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=32 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:43:49.883430 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:43:49.895617 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd5f6b-3557-4331-8176-3669aae14e14 service nova] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:43:49.899720 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:43:49.900466 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:43:49.900828 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:43:49.901218 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:43:49.903542 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:43:49.919770 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:43:49.921515 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:43:49.933821 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1483/2940] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:43:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:43:51.127993 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:43:51.136744 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f0fbc4-bd3f-42ae-8a09-f5c0bcaa0e62 service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:43:51.140670 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:43:51.141388 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:43:51.141659 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:43:51.142013 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:43:51.143884 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:43:51.157877 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:43:51.159306 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:43:51.170547 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1458/2941] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:43:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:43:59.928401 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=6 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:44:01.165890 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=6 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:44:02.346727 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:44:02.348225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:44:02.360109 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccbad5a-5c9e-470f-aa9f-8f772cfc0db9 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:44:02.364368 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '56aaa029a9c542e0bc4658c5cd55c2b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:44:02.365004 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:44:02.365313 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:44:02.365582 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:44:02.367578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:44:02.445945 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:44:02.447406 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:44:02.457957 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1484/2942] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:44:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:44:09.305041 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:44:09.307822 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:44:09.325483 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda0b4b0-a09e-4d84-80c6-d99dedab68ec service neutron] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:44:09.331032 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:44:09.331677 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:44:09.331953 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:44:09.332809 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] PATH_INFO: `/v3/users/3fe5a1a037fa460eb3121e4d307542da` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:44:09.333404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fe5a1a037fa460eb3121e4d307542da)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:44:09.335513 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:44:09.397808 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fe5a1a037fa460eb3121e4d307542da'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.413484 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fe5a1a037fa460eb3121e4d307542da'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.421300 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fe5a1a037fa460eb3121e4d307542da'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.421949 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fe5a1a037fa460eb3121e4d307542da'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.422558 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fe5a1a037fa460eb3121e4d307542da'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.440383 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1459/2943] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:44:09 2022] DELETE /identity/v3/users/3fe5a1a037fa460eb3121e4d307542da => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:44:09.632697 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:44:09.643061 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690e34-1196-43e1-919a-ec1c1ae3a833 service neutron] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:44:09.700827 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:44:09.701493 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:44:09.701766 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:44:09.702032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] PATH_INFO: `/v3/projects/79e671941c234f36b542080d256de2c6` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:44:09.703167 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79e671941c234f36b542080d256de2c6)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:44:09.705058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:44:09.752883 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79e671941c234f36b542080d256de2c6', 'request_id': 'req-9f762918-0061-4b81-b939-a6d80b347096'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.759128 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79e671941c234f36b542080d256de2c6', 'request_id': 'req-9f762918-0061-4b81-b939-a6d80b347096'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.764920 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79e671941c234f36b542080d256de2c6', 'request_id': 'req-9f762918-0061-4b81-b939-a6d80b347096'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.777961 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] The token cache is being invalidate because project 79e671941c234f36b542080d256de2c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:44:09.778246 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.778857 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.780317 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1485/2944] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:44:09 2022] DELETE /identity/v3/projects/79e671941c234f36b542080d256de2c6 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:44:09.784404 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:44:09.785836 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:44:09.798920 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5ef272-0f73-42ba-a0a5-6ce38fc7c296 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:44:09.857554 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:44:09.858180 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:44:09.858453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:44:09.858719 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] PATH_INFO: `/v3/users/d8285f30bcee4d408d40a243f66a4b15` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:44:09.859960 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8285f30bcee4d408d40a243f66a4b15)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:44:09.861896 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:44:09.908296 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8285f30bcee4d408d40a243f66a4b15'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.914950 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8285f30bcee4d408d40a243f66a4b15'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.923498 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8285f30bcee4d408d40a243f66a4b15'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.924226 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8285f30bcee4d408d40a243f66a4b15'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.924886 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-54b57ee4-ba05-48ce-a6ab-d90409315094 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8285f30bcee4d408d40a243f66a4b15'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:09.933269 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:44:09.944826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1460/2945] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:44:09 2022] DELETE /identity/v3/users/d8285f30bcee4d408d40a243f66a4b15 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:44:10.126935 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:44:10.128428 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:44:10.147362 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f762918-0061-4b81-b939-a6d80b347096 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:44:10.170325 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:44:10.209420 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:44:10.210058 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:44:10.210316 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:44:10.210603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] PATH_INFO: `/v3/projects/6e8c33b5be4d43799914bbc93fe5954f` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:44:10.211730 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e8c33b5be4d43799914bbc93fe5954f)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:44:10.213653 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:44:10.261506 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e8c33b5be4d43799914bbc93fe5954f', 'request_id': 'req-6202ac0a-ccb9-4c17-84c5-516ca939ed14'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:10.268096 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e8c33b5be4d43799914bbc93fe5954f', 'request_id': 'req-6202ac0a-ccb9-4c17-84c5-516ca939ed14'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:10.273490 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e8c33b5be4d43799914bbc93fe5954f', 'request_id': 'req-6202ac0a-ccb9-4c17-84c5-516ca939ed14'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:10.294815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] The token cache is being invalidate because project 6e8c33b5be4d43799914bbc93fe5954f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:44:10.295143 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:10.296146 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:44:10.299998 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1486/2946] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:44:10 2022] DELETE /identity/v3/projects/6e8c33b5be4d43799914bbc93fe5954f => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:44:19.945626 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=3,keystone:DELETE=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:44:20.290122 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:46:08.320453 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] REQUEST_METHOD: `POST` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:46:08.321067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:46:08.321067 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:46:08.385256 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.auth.core [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] MFA Rules not processed for user `1a14c9fa11f94a3ab0c4d32f6f8f0931`. Rule list: `[]` (Enabled: `True`). {{(pid=81746) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 27 02:46:08.386732 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:46:08.466687 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1461/2947] 10.222.0.52 () {58 vars in 967 bytes} [Sat Aug 27 02:46:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 27 02:46:08.511775 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:46:08.513481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:46:08.614522 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6202ac0a-ccb9-4c17-84c5-516ca939ed14 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:46:08.621295 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9a334f972474aa19aa3beae51560f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:46:08.621896 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] REQUEST_METHOD: `GET` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:46:08.622153 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:46:08.622412 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:46:08.631586 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:46:08.633212 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:46:08.642259 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1487/2948] 10.222.0.52 () {60 vars in 1384 bytes} [Sat Aug 27 02:46:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 27 02:46:15.909612 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:46:15.911989 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:46:15.991690 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e1cc59-3b01-4ef7-88e8-dda2a97e1e9e None None] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:46:15.996593 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:46:15.997260 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:46:15.997555 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:46:15.997806 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] PATH_INFO: `/v3/users/ba73716667184588b522bc7856404be8` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:46:15.999121 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba73716667184588b522bc7856404be8)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:46:16.001062 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:46:16.056826 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba73716667184588b522bc7856404be8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.065335 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba73716667184588b522bc7856404be8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.074028 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba73716667184588b522bc7856404be8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.074888 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba73716667184588b522bc7856404be8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.075492 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba73716667184588b522bc7856404be8'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.077844 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:46:16.097163 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1462/2949] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:46:15 2022] DELETE /identity/v3/users/ba73716667184588b522bc7856404be8 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:46:16.295558 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:46:16.306745 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd455cc0-2b27-4d56-86c8-d646113f1f60 service swift] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:46:16.373185 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:46:16.373815 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:46:16.374085 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:46:16.374351 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] PATH_INFO: `/v3/projects/7dd5f20c4a714ee3b555d4aef8b87730` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:46:16.375488 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7dd5f20c4a714ee3b555d4aef8b87730)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:46:16.377413 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:46:16.433525 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7dd5f20c4a714ee3b555d4aef8b87730', 'request_id': 'req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.441749 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7dd5f20c4a714ee3b555d4aef8b87730', 'request_id': 'req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.448600 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7dd5f20c4a714ee3b555d4aef8b87730', 'request_id': 'req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.460445 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:46:16.467841 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] The token cache is being invalidate because project 7dd5f20c4a714ee3b555d4aef8b87730 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:46:16.468358 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.469473 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.471334 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1488/2950] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:46:16 2022] DELETE /identity/v3/projects/7dd5f20c4a714ee3b555d4aef8b87730 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:46:16.475515 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:46:16.477036 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:46:16.491165 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f0d4de-f4b7-4988-a234-d9879169f9dc admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:46:16.557344 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:46:16.557973 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:46:16.558752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:46:16.558752 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] PATH_INFO: `/v3/users/eab0409bbbbb4b39a0fc34421cfdd377` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:46:16.559864 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eab0409bbbbb4b39a0fc34421cfdd377)` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:46:16.561847 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:46:16.611701 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eab0409bbbbb4b39a0fc34421cfdd377'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.618566 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eab0409bbbbb4b39a0fc34421cfdd377'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.626691 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eab0409bbbbb4b39a0fc34421cfdd377'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.627999 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eab0409bbbbb4b39a0fc34421cfdd377'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.629166 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.notifications [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eab0409bbbbb4b39a0fc34421cfdd377'} {{(pid=81746) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:16.649461 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1463/2951] 10.222.0.52 () {58 vars in 1224 bytes} [Sat Aug 27 02:46:16 2022] DELETE /identity/v3/users/eab0409bbbbb4b39a0fc34421cfdd377 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:46:16.848685 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Authenticating user token {{(pid=81745) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:46:16.850860 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.fernet_utils [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81745) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:46:16.874225 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173d04c-d63d-4b1b-b5f5-ac4f952dc4c6 admin admin] Validating token access rules against request {{(pid=81745) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:46:16.953199 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6648709a4b6d470c80e9b42cb0d345ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1df9f889b3642a7ac056884bc2c51a4', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81745) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:46:16.953754 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:46:16.954032 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] SCRIPT_NAME: `/identity` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:46:16.954345 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] PATH_INFO: `/v3/projects/71573f3978164a2894f5b9632e0722de` {{(pid=81745) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:46:16.955578 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] RBAC: Authorizing `identity:delete_project(project_id=71573f3978164a2894f5b9632e0722de)` {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:46:16.957603 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] RBAC: Authorization granted {{(pid=81745) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:46:17.019249 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71573f3978164a2894f5b9632e0722de', 'request_id': 'req-ef85ecac-4b8f-4605-96b4-7c8de897c821'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:17.025987 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71573f3978164a2894f5b9632e0722de', 'request_id': 'req-ef85ecac-4b8f-4605-96b4-7c8de897c821'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:17.031903 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71573f3978164a2894f5b9632e0722de', 'request_id': 'req-ef85ecac-4b8f-4605-96b4-7c8de897c821'} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:17.045481 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] The token cache is being invalidate because project 71573f3978164a2894f5b9632e0722de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81745) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 27 02:46:17.045750 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:17.046480 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG keystone.notifications [None req-ef85ecac-4b8f-4605-96b4-7c8de897c821 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81745) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 27 02:46:17.048031 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: [pid: 81745|app: 0|req: 1489/2952] 10.222.0.52 () {58 vars in 1233 bytes} [Sat Aug 27 02:46:16 2022] DELETE /identity/v3/projects/71573f3978164a2894f5b9632e0722de => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 27 02:46:26.645668 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=59,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:46:27.042449 dv-cinder-846303-3-iscsi devstack@keystone.service[81745]: DEBUG dbcounter [-] [81745] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81745) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 27 02:46:57.310759 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Authenticating user token {{(pid=81746) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 27 02:46:57.312421 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:46:57.387278 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadad1a8-f5b4-4d23-80dc-7ffedd9ea371 admin admin] Validating token access rules against request {{(pid=81746) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 27 02:46:57.457812 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665edf3e-b42e-4648-b20e-2d00e1ce1941 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa1922df0af7489db403cb6e42a7311b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '89716fa846034fe3b8d98abb238e5f22', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81746) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 27 02:46:57.458325 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665edf3e-b42e-4648-b20e-2d00e1ce1941 service placement] REQUEST_METHOD: `GET` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 27 02:46:57.458603 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665edf3e-b42e-4648-b20e-2d00e1ce1941 service placement] SCRIPT_NAME: `/identity` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 27 02:46:57.458869 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665edf3e-b42e-4648-b20e-2d00e1ce1941 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81746) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 27 02:46:57.460698 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.fernet_utils [None req-665edf3e-b42e-4648-b20e-2d00e1ce1941 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81746) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 27 02:46:57.476294 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665edf3e-b42e-4648-b20e-2d00e1ce1941 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 27 02:46:57.478657 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665edf3e-b42e-4648-b20e-2d00e1ce1941 service placement] RBAC: Authorization granted {{(pid=81746) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 27 02:46:57.493080 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: [pid: 81746|app: 0|req: 1464/2953] 10.222.0.52 () {60 vars in 1365 bytes} [Sat Aug 27 02:46:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 27 02:47:07.486790 dv-cinder-846303-3-iscsi devstack@keystone.service[81746]: DEBUG dbcounter [-] [81746] Writing DB stats keystone:SELECT=58 {{(pid=81746) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}